2023-09-29T21:56:54.842 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T21:56:54.843 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} --owner scheduled_yuriw@teuthology --verbose --name yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303 -- /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/orig.config.yaml 2023-09-29T21:56:54.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T21:56:54.946 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303 branch: wip-yuri11-testing-2023-09-28-1611-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7406303' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri11-testing-2023-09-28-1611-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: wip-yuri11-testing-2023-09-28-1611-reef sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4212 sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-yuri11-testing-2023-09-28-1611-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 targets: smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGg1Rj0/eERk8ONdHWi2h+CdqkStQ5MR5QKLDJW4g0aqKq3EScLjXaZ/6ZMMiGzDXOObRTOT0dqrZkf9wcsjTQk= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjK/BJMklETggsMW2i4eJ/4RHkZRjVcDsj06GMnEus1nm4X8DpnPcTSVd41NffPZrjXOA+ewpgniZENFjXO//c= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_12:36:11 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T21:56:54.947 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa; will attempt to use it 2023-09-29T21:56:54.947 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks 2023-09-29T21:56:54.948 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T21:56:54.949 INFO:teuthology.task.internal:Checking packages... 2023-09-29T21:56:54.971 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '38c3646d535f6f917a872ef9baef468589ba9f32' 2023-09-29T21:56:54.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T21:56:54.972 INFO:teuthology.packaging:ref: None 2023-09-29T21:56:54.972 INFO:teuthology.packaging:tag: None 2023-09-29T21:56:54.972 INFO:teuthology.packaging:branch: wip-yuri11-testing-2023-09-28-1611-reef 2023-09-29T21:56:54.972 INFO:teuthology.packaging:sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:56:54.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri11-testing-2023-09-28-1611-reef 2023-09-29T21:56:55.119 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-524.g38c3646d 2023-09-29T21:56:55.119 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T21:56:55.126 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T21:56:55.126 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T21:56:55.170 INFO:teuthology.task.internal:Saving configuration 2023-09-29T21:56:55.184 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T21:56:55.203 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T21:56:55.226 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:51:56.285915', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGg1Rj0/eERk8ONdHWi2h+CdqkStQ5MR5QKLDJW4g0aqKq3EScLjXaZ/6ZMMiGzDXOObRTOT0dqrZkf9wcsjTQk='} 2023-09-29T21:56:55.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:51:56.284120', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjK/BJMklETggsMW2i4eJ/4RHkZRjVcDsj06GMnEus1nm4X8DpnPcTSVd41NffPZrjXOA+ewpgniZENFjXO//c='} 2023-09-29T21:56:55.255 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T21:56:55.261 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-09-29T21:56:55.262 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-09-29T21:56:55.262 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T21:56:55.382 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc714d9fee0>, signals=[15]) 2023-09-29T21:56:55.383 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T21:56:55.388 INFO:teuthology.task.internal:Opening connections... 2023-09-29T21:56:55.389 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T21:56:55.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:56:55.480 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T21:56:55.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:56:55.564 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T21:56:55.570 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2023-09-29T21:56:55.589 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2023-09-29T21:56:55.589 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2023-09-29T21:56:55.649 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Stream" 2023-09-29T21:56:55.649 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="9" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="9" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el9" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Stream 9" 2023-09-29T21:56:55.650 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:LOGO="fedora-logo-icon" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-09-29T21:56:55.651 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T21:56:55.652 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2023-09-29T21:56:55.676 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2023-09-29T21:56:55.693 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2023-09-29T21:56:55.693 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2023-09-29T21:56:55.751 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-09-29T21:56:55.752 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T21:56:55.752 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-09-29T21:56:55.775 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T21:56:55.784 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T21:56:55.828 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T21:56:55.829 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2023-09-29T21:56:55.831 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2023-09-29T21:56:55.847 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T21:56:55.870 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T21:56:55.870 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2023-09-29T21:56:55.892 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2023-09-29T21:56:56.013 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T21:56:56.065 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T21:56:56.079 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T21:56:56.079 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T21:56:56.079 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T21:56:56.080 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T21:56:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:56:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:56:56.097 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2023-09-29T21:56:56.097 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2023-09-29T21:56:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-361.el9.x86_64 2023-09-29T21:56:56.154 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-361.el9.x86_64 2023-09-29T21:56:56.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2023-09-29T21:56:56.157 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-361.el9.x86_64 2023-09-29T21:56:56.157 INFO:teuthology.task.kernel:Running kernel on smithi175: 5.14.0-361.el9.x86_64 2023-09-29T21:56:56.157 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2023-09-29T21:56:58.259 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-09-29T21:56:58.297 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-09-29T21:56:59.635 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-09-29T21:56:59.664 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-09-29T21:57:00.884 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 7.8 MB 00:00 2023-09-29T21:57:04.879 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 7.8 MB 00:04 2023-09-29T21:57:05.025 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 18 MB 00:03 2023-09-29T21:57:07.882 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 18 MB 00:02 2023-09-29T21:57:09.218 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 5.7 MB 00:00 2023-09-29T21:57:10.686 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 14 kB 00:00 2023-09-29T21:57:11.605 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.7 MB 00:00 2023-09-29T21:57:13.097 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 14 kB 00:00 2023-09-29T21:57:14.226 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 18 MB/s | 19 MB 00:01 2023-09-29T21:57:15.106 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 19 MB 00:04 2023-09-29T21:57:18.631 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2023-09-29T21:57:19.503 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2023-09-29T21:57:20.572 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-09-29T21:57:20.572 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-09-29T21:57:20.666 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: kernel x86_64 5.14.0-368.el9 baseos 5.1 M 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core x86_64 5.14.0-368.el9 baseos 20 M 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules x86_64 5.14.0-368.el9 baseos 37 M 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core x86_64 5.14.0-368.el9 baseos 32 M 2023-09-29T21:57:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 94 M 2023-09-29T21:57:20.683 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 120 M 2023-09-29T21:57:20.684 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T21:57:21.435 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-09-29T21:57:21.435 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-09-29T21:57:21.441 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): kernel-5.14.0-368.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2023-09-29T21:57:21.524 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T21:57:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T21:57:21.534 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-09-29T21:57:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T21:57:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T21:57:21.534 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-368.el9 baseos 5.1 M 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-368.el9 baseos 20 M 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-368.el9 baseos 37 M 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-368.el9 baseos 32 M 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2023-09-29T21:57:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:57:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 94 M 2023-09-29T21:57:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 120 M 2023-09-29T21:57:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T21:57:22.230 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-368.el9.x86_64.rpm 9.1 MB/s | 5.1 MB 00:00 2023-09-29T21:57:22.383 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): kernel-core-5.14.0-368.el9.x86_64.rpm 14 MB/s | 20 MB 00:01 2023-09-29T21:57:22.758 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): kernel-modules-core-5.14.0-368.el9.x86_6 24 MB/s | 32 MB 00:01 2023-09-29T21:57:22.838 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-368.el9.x86_64.rpm 17 MB/s | 20 MB 00:01 2023-09-29T21:57:23.480 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-5.14.0-368.el9.x86_64.rpm 21 MB/s | 37 MB 00:01 2023-09-29T21:57:23.763 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-core-5.14.0-368.el9.x86_6 21 MB/s | 32 MB 00:01 2023-09-29T21:57:23.764 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T21:57:23.764 INFO:teuthology.orchestra.run.smithi184.stdout:Total 42 MB/s | 94 MB 00:02 2023-09-29T21:57:24.591 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): kernel-modules-5.14.0-368.el9.x86_64.rpm 10 MB/s | 37 MB 00:03 2023-09-29T21:57:24.592 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T21:57:24.592 INFO:teuthology.orchestra.run.smithi175.stdout:Total 24 MB/s | 94 MB 00:03 2023-09-29T21:57:24.594 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T21:57:24.925 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T21:57:24.925 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T21:57:25.431 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T21:57:25.780 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T21:57:25.781 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T21:57:26.228 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T21:57:26.229 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T21:57:27.063 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T21:57:27.064 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T21:57:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T21:57:29.462 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-368.el9.x86_64 1/4 2023-09-29T21:57:29.614 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:57:29.957 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T21:57:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-core-5.14.0-368.el9.x86_64 1/4 2023-09-29T21:57:30.699 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:57:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:57:32.518 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:57:33.439 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:57:33.616 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:57:37.079 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:57:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:57:38.215 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:57:38.504 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:57:41.940 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:57:43.088 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:10.789 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:10.831 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:11.231 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:11.231 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-368.el9.x86_64 1/4 2023-09-29T21:58:11.231 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:58:11.231 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:58:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-368.el9.x86_64 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-368.el9.x86_64 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-368.el9.x86_64 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:58:11.649 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T21:58:11.811 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:11.854 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:12.132 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T21:58:12.254 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:12.254 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-5.14.0-368.el9.x86_64 1/4 2023-09-29T21:58:12.254 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-core-5.14.0-368.el9.x86_64 2/4 2023-09-29T21:58:12.254 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-5.14.0-368.el9.x86_64 3/4 2023-09-29T21:58:12.311 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.312 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-368.el9.x86_64 2023-09-29T21:58:12.312 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-368.el9.x86_64 2023-09-29T21:58:12.312 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T21:58:12.312 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T21:58:12.312 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-09-29T21:58:12.312 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T21:58:12.313 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T21:58:12.427 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.427 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-368.el9.x86_64 2023-09-29T21:58:12.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2023-09-29T21:58:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2023-09-29T21:58:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2023-09-29T21:58:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2023-09-29T21:58:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2023-09-29T21:58:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T21:58:12.472 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T21:58:12.473 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T21:58:12.474 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T21:58:12.474 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-368.el9.x86_64 2023-09-29T21:58:12.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-core-5.14.0-368.el9.x86_64 4/4 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core-5.14.0-368.el9.x86_64 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:58:12.696 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T21:58:13.076 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2023-09-29T21:58:13.150 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T21:58:13.298 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:13.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-368.el9.x86_64 2023-09-29T21:58:13.298 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-368.el9.x86_64 2023-09-29T21:58:13.298 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T21:58:13.298 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T21:58:13.299 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-09-29T21:58:13.299 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T21:58:13.299 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T21:58:13.414 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-368.el9.x86_64 2023-09-29T21:58:13.414 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-368.el9.x86_64 2023-09-29T21:58:13.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2023-09-29T21:58:13.456 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.06 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 68.el9 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Group : Unspecified 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 8264254 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-09-29T21:58:13.457 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : builder@centos.org 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : CentOS 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2023-09-29T21:58:13.458 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T21:58:13.459 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T21:58:13.459 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T21:58:13.459 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 5.14.0-368.el9.x86_64 2023-09-29T21:58:13.460 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T21:58:14.077 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2023-09-29T21:58:14.487 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-09-29T21:58:14.530 INFO:teuthology.orchestra.run.smithi184.stderr:done 2023-09-29T21:58:14.533 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2023-09-29T21:58:14.548 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.Yh0hP4IIXF 2023-09-29T21:58:14.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Yh0hP4IIXF 2023-09-29T21:58:14.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.Yh0hP4IIXF 2023-09-29T21:58:14.756 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.Yh0hP4IIXF is 6KB 2023-09-29T21:58:14.804 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.Yh0hP4IIXF 2023-09-29T21:58:14.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T21:58:14.884 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-09-29T21:58:14.885 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-09-29T21:58:14.885 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-09-29T21:58:14.885 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-368.el9.x86_64.conf 2023-09-29T21:58:14.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-368.el9.x86_64 2023-09-29T21:58:15.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2023-09-29T21:58:15.522 INFO:teuthology.orchestra.run.smithi175.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-09-29T21:58:15.552 INFO:teuthology.orchestra.run.smithi175.stderr:done 2023-09-29T21:58:15.554 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2023-09-29T21:58:15.569 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.iUpOpexlOe 2023-09-29T21:58:15.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iUpOpexlOe 2023-09-29T21:58:15.638 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.iUpOpexlOe 2023-09-29T21:58:15.782 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.iUpOpexlOe is 6KB 2023-09-29T21:58:15.830 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.iUpOpexlOe 2023-09-29T21:58:15.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T21:58:15.909 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-09-29T21:58:15.909 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-09-29T21:58:15.909 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-09-29T21:58:15.909 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-368.el9.x86_64.conf 2023-09-29T21:58:15.911 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-368.el9.x86_64 2023-09-29T21:58:16.054 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2023-09-29T21:58:45.070 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-09-29T21:58:45.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-09-29T21:58:45.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:58:46.057 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-09-29T21:58:46.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-09-29T21:58:46.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:59:45.131 DEBUG:teuthology.orchestra.remote:timed out 2023-09-29T21:59:46.061 DEBUG:teuthology.orchestra.remote:timed out 2023-09-29T21:59:54.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-09-29T21:59:54.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:59:54.497 DEBUG:teuthology.orchestra.run.smithi184:> true 2023-09-29T21:59:54.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-09-29T21:59:54.866 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.14.0-368.el9.x86_64"... 2023-09-29T21:59:54.867 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2023-09-29T21:59:54.880 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-368.el9.x86_64 2023-09-29T21:59:54.880 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-368.el9.x86_64 vs 5.14.0-368.el9.x86_64 2023-09-29T21:59:54.881 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T21:59:54.881 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T21:59:55.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-09-29T21:59:55.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:59:55.881 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-09-29T21:59:55.881 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T21:59:55.974 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2023-09-29T21:59:56.021 DEBUG:teuthology.parallel:result is None 2023-09-29T21:59:56.539 DEBUG:teuthology.orchestra.run.smithi175:> true 2023-09-29T21:59:56.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-09-29T21:59:56.967 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.14.0-368.el9.x86_64"... 2023-09-29T21:59:56.967 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2023-09-29T21:59:56.980 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-368.el9.x86_64 2023-09-29T21:59:56.980 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-368.el9.x86_64 vs 5.14.0-368.el9.x86_64 2023-09-29T21:59:56.981 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T21:59:56.981 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T21:59:57.982 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-09-29T21:59:57.982 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T21:59:58.091 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2023-09-29T21:59:58.131 DEBUG:teuthology.parallel:result is None 2023-09-29T21:59:58.131 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T21:59:58.139 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T21:59:58.139 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T21:59:58.141 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T21:59:58.165 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T21:59:58.172 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T21:59:58.202 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T21:59:58.203 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T21:59:58.207 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T21:59:58.248 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T21:59:58.259 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T21:59:58.259 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T21:59:58.270 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T21:59:58.300 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:59:58.313 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:59:58.342 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:59:58.371 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:59:58.373 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T21:59:58.380 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T21:59:58.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T21:59:58.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T21:59:58.426 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T21:59:58.438 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T21:59:58.438 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T21:59:58.449 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T21:59:58.490 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:59:58.543 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:59:58.602 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:59:58.684 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:59:58.725 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T21:59:58.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T21:59:58.793 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:59:58.835 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:59:59.004 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:59:59.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:59:59.088 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T21:59:59.088 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T21:59:59.157 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2023-09-29T21:59:59.159 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2023-09-29T21:59:59.204 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T21:59:59.248 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T21:59:59.474 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T21:59:59.483 INFO:teuthology.task.internal:Starting timer... 2023-09-29T21:59:59.483 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T21:59:59.557 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T21:59:59.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2023-09-29T21:59:59.725 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: 2023-09-29T21:59:59.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2023-09-29T21:59:59.790 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2023-09-29T21:59:59.791 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T21:59:59.791 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2023-09-29T21:59:59.836 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2023-09-29T21:59:59.836 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2023-09-29T21:59:59.870 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2023-09-29T21:59:59.870 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi175.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2023-09-29T21:59:59.870 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T21:59:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:59:59.897 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T21:59:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:59:59.938 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T21:59:59.938 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2023-09-29T21:59:59.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2023-09-29T22:00:00.014 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T22:00:00.023 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-29T22:00:00.023 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T22:00:00.040 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T22:00:00.041 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi175.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T22:05:12.991 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi175.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2023-09-29T22:05:12.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-09-29T22:05:12.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:05:13.079 DEBUG:teuthology.orchestra.run.smithi175:> true 2023-09-29T22:05:13.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-09-29T22:05:13.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-09-29T22:05:13.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:05:13.229 DEBUG:teuthology.orchestra.run.smithi184:> true 2023-09-29T22:05:13.297 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-09-29T22:05:13.297 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T22:05:13.311 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T22:05:13.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T22:05:13.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T22:05:13.314 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T22:05:13.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T22:05:13.342 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T22:05:13.359 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T22:05:13.369 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T22:05:13.386 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T22:05:13.387 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2023-09-29T22:05:13.401 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2023-09-29T22:05:13.415 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2023-09-29T22:05:13.417 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T22:05:13.429 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2023-09-29T22:05:13.433 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T22:05:13.445 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T22:05:13.461 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T22:05:13.477 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2023-09-29T22:05:13.480 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T22:05:13.480 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2023-09-29T22:05:13.481 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.481 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.481 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.481 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.504 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.508 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T22:05:13.509 INFO:teuthology.run_tasks:Running task install... 2023-09-29T22:05:13.517 DEBUG:teuthology.task.install:project ceph 2023-09-29T22:05:13.517 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}} 2023-09-29T22:05:13.518 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'} 2023-09-29T22:05:13.518 INFO:teuthology.task.install:Using flavor: default 2023-09-29T22:05:13.523 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-29T22:05:13.523 INFO:teuthology.task.install:extra packages: [] 2023-09-29T22:05:13.524 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'tag': None, 'wait_for_package': False} 2023-09-29T22:05:13.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T22:05:13.526 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'tag': None, 'wait_for_package': False} 2023-09-29T22:05:13.526 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T22:05:13.664 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/ 2023-09-29T22:05:13.664 INFO:teuthology.task.install.rpm:Package version is 18.2.0-524.g38c3646d 2023-09-29T22:05:13.742 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/ 2023-09-29T22:05:13.742 INFO:teuthology.task.install.rpm:Package version is 18.2.0-524.g38c3646d 2023-09-29T22:05:13.782 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T22:05:13.782 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:05:13.782 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T22:05:13.811 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T22:05:13.811 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38c3646d535f6f917a872ef9baef468589ba9f32/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T22:05:13.852 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T22:05:13.853 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:05:13.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T22:05:13.880 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T22:05:13.881 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T22:05:13.881 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38c3646d535f6f917a872ef9baef468589ba9f32/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T22:05:13.948 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T22:05:13.959 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T22:05:14.025 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2023-09-29T22:05:14.027 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2023-09-29T22:05:14.028 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T22:05:14.095 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2023-09-29T22:05:14.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2023-09-29T22:05:14.262 INFO:teuthology.orchestra.run.smithi175.stdout:69 files removed 2023-09-29T22:05:14.290 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw 2023-09-29T22:05:14.332 INFO:teuthology.orchestra.run.smithi184.stdout:69 files removed 2023-09-29T22:05:14.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2023-09-29T22:05:14.994 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 199 kB/s | 77 kB 00:00 2023-09-29T22:05:15.113 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 175 kB/s | 77 kB 00:00 2023-09-29T22:05:15.327 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-09-29T22:05:15.489 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-09-29T22:05:15.670 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2023-09-29T22:05:15.906 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2023-09-29T22:05:16.289 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - AppStream 11 MB/s | 6.3 MB 00:00 2023-09-29T22:05:17.058 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 5.6 MB/s | 6.3 MB 00:01 2023-09-29T22:05:17.639 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-09-29T22:05:18.375 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-09-29T22:05:18.955 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 7.8 MB 00:00 2023-09-29T22:05:19.350 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 7.8 MB 00:00 2023-09-29T22:05:21.640 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-09-29T22:05:22.309 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 18 MB 00:01 2023-09-29T22:05:26.284 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.7 MB 00:01 2023-09-29T22:05:26.851 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 5.7 MB 00:01 2023-09-29T22:05:27.591 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 14 kB 00:00 2023-09-29T22:05:28.526 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 22 MB/s | 19 MB 00:00 2023-09-29T22:05:28.986 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 14 kB 00:00 2023-09-29T22:05:31.095 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 19 MB 00:02 2023-09-29T22:05:32.917 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2023-09-29T22:05:35.044 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:18.2.0-524.g38c3646d.el9 ceph 7.7 M 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2023-09-29T22:05:35.048 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.3 M 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.0 M 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:18.2.0-524.g38c3646d.el9 ceph 5.2 M 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:18.2.0-524.g38c3646d.el9 ceph 18 M 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:18.2.0-524.g38c3646d.el9 ceph 24 k 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-09-29T22:05:35.049 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 654 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 474 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 4.4 M 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 45 k 2023-09-29T22:05:35.050 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:18.2.0-524.g38c3646d.el9 ceph 124 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:18.2.0-524.g38c3646d.el9 ceph 156 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:18.2.0-524.g38c3646d.el9 ceph 321 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 298 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:18.2.0-524.g38c3646d.el9 ceph 99 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout:Install 28 Packages 2023-09-29T22:05:35.051 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2023-09-29T22:05:35.052 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:35.052 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 53 M 2023-09-29T22:05:35.052 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:05:35.487 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-09-29T22:05:36.052 INFO:teuthology.orchestra.run.smithi175.stdout:(1/30): ceph-base-18.2.0-524.g38c3646d.el9.x86_ 7.8 MB/s | 5.2 MB 00:00 2023-09-29T22:05:36.102 INFO:teuthology.orchestra.run.smithi175.stdout:(2/30): ceph-selinux-18.2.0-524.g38c3646d.el9.x 482 kB/s | 24 kB 00:00 2023-09-29T22:05:36.186 INFO:teuthology.orchestra.run.smithi175.stdout:(3/30): libcephfs2-18.2.0-524.g38c3646d.el9.x86 7.7 MB/s | 654 kB 00:00 2023-09-29T22:05:36.253 INFO:teuthology.orchestra.run.smithi175.stdout:(4/30): libradosstriper1-18.2.0-524.g38c3646d.e 6.9 MB/s | 474 kB 00:00 2023-09-29T22:05:36.645 INFO:teuthology.orchestra.run.smithi175.stdout:(5/30): librgw2-18.2.0-524.g38c3646d.el9.x86_64 11 MB/s | 4.4 MB 00:00 2023-09-29T22:05:36.695 INFO:teuthology.orchestra.run.smithi175.stdout:(6/30): python3-ceph-argparse-18.2.0-524.g38c36 904 kB/s | 45 kB 00:00 2023-09-29T22:05:36.745 INFO:teuthology.orchestra.run.smithi175.stdout:(7/30): python3-ceph-common-18.2.0-524.g38c3646 2.4 MB/s | 124 kB 00:00 2023-09-29T22:05:36.829 INFO:teuthology.orchestra.run.smithi175.stdout:(8/30): ceph-radosgw-18.2.0-524.g38c3646d.el9.x 5.3 MB/s | 7.7 MB 00:01 2023-09-29T22:05:36.854 INFO:teuthology.orchestra.run.smithi175.stdout:(9/30): python3-cephfs-18.2.0-524.g38c3646d.el9 1.4 MB/s | 156 kB 00:00 2023-09-29T22:05:36.880 INFO:teuthology.orchestra.run.smithi175.stdout:(10/30): python3-rados-18.2.0-524.g38c3646d.el9 6.3 MB/s | 321 kB 00:00 2023-09-29T22:05:36.913 INFO:teuthology.orchestra.run.smithi175.stdout:(11/30): python3-rbd-18.2.0-524.g38c3646d.el9.x 5.0 MB/s | 298 kB 00:00 2023-09-29T22:05:36.947 INFO:teuthology.orchestra.run.smithi175.stdout:(12/30): python3-rgw-18.2.0-524.g38c3646d.el9.x 1.5 MB/s | 99 kB 00:00 2023-09-29T22:05:37.064 INFO:teuthology.orchestra.run.smithi175.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2023-09-29T22:05:37.105 INFO:teuthology.orchestra.run.smithi175.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-09-29T22:05:37.148 INFO:teuthology.orchestra.run.smithi175.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2023-09-29T22:05:37.173 INFO:teuthology.orchestra.run.smithi175.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-09-29T22:05:37.198 INFO:teuthology.orchestra.run.smithi175.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2023-09-29T22:05:37.290 INFO:teuthology.orchestra.run.smithi175.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2023-09-29T22:05:37.349 INFO:teuthology.orchestra.run.smithi175.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 737 kB/s | 111 kB 00:00 2023-09-29T22:05:37.465 INFO:teuthology.orchestra.run.smithi175.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2023-09-29T22:05:37.549 INFO:teuthology.orchestra.run.smithi175.stdout:(21/30): libarrow-9.0.0-9.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2023-09-29T22:05:37.566 INFO:teuthology.orchestra.run.smithi175.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 242 kB/s | 24 kB 00:00 2023-09-29T22:05:37.599 INFO:teuthology.orchestra.run.smithi175.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2023-09-29T22:05:37.633 INFO:teuthology.orchestra.run.smithi175.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-09-29T22:05:37.635 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:18.2.0-524.g38c3646d.el9 ceph 7.7 M 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2023-09-29T22:05:37.649 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.3 M 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.0 M 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:18.2.0-524.g38c3646d.el9 ceph 5.2 M 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:18.2.0-524.g38c3646d.el9 ceph 18 M 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:18.2.0-524.g38c3646d.el9 ceph 24 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 654 k 2023-09-29T22:05:37.650 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 474 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:18.2.0-524.g38c3646d.el9 ceph 4.4 M 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 45 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:18.2.0-524.g38c3646d.el9 ceph 124 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:18.2.0-524.g38c3646d.el9 ceph 156 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-09-29T22:05:37.651 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:18.2.0-524.g38c3646d.el9 ceph 321 k 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 298 k 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:18.2.0-524.g38c3646d.el9 ceph 99 k 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout:Install 28 Packages 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2023-09-29T22:05:37.652 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:05:37.653 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 53 M 2023-09-29T22:05:37.653 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:05:37.709 INFO:teuthology.orchestra.run.smithi175.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 7.5 MB/s | 837 kB 00:00 2023-09-29T22:05:37.751 INFO:teuthology.orchestra.run.smithi175.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2023-09-29T22:05:37.843 INFO:teuthology.orchestra.run.smithi175.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-09-29T22:05:38.085 INFO:teuthology.orchestra.run.smithi175.stdout:(28/30): librados2-18.2.0-524.g38c3646d.el9.x86 9.8 MB/s | 3.3 MB 00:00 2023-09-29T22:05:38.318 INFO:teuthology.orchestra.run.smithi175.stdout:(29/30): ceph-common-18.2.0-524.g38c3646d.el9.x 6.2 MB/s | 18 MB 00:02 2023-09-29T22:05:38.485 INFO:teuthology.orchestra.run.smithi175.stdout:(30/30): librbd1-18.2.0-524.g38c3646d.el9.x86_6 4.7 MB/s | 3.0 MB 00:00 2023-09-29T22:05:38.489 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:05:38.489 INFO:teuthology.orchestra.run.smithi175.stdout:Total 15 MB/s | 53 MB 00:03 2023-09-29T22:05:38.499 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:05:38.533 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:05:38.533 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:05:38.638 INFO:teuthology.orchestra.run.smithi184.stdout:(1/30): ceph-base-18.2.0-524.g38c3646d.el9.x86_ 8.6 MB/s | 5.2 MB 00:00 2023-09-29T22:05:38.688 INFO:teuthology.orchestra.run.smithi184.stdout:(2/30): ceph-selinux-18.2.0-524.g38c3646d.el9.x 484 kB/s | 24 kB 00:00 2023-09-29T22:05:38.763 INFO:teuthology.orchestra.run.smithi184.stdout:(3/30): libcephfs2-18.2.0-524.g38c3646d.el9.x86 8.5 MB/s | 654 kB 00:00 2023-09-29T22:05:38.830 INFO:teuthology.orchestra.run.smithi184.stdout:(4/30): libradosstriper1-18.2.0-524.g38c3646d.e 6.9 MB/s | 474 kB 00:00 2023-09-29T22:05:39.047 INFO:teuthology.orchestra.run.smithi184.stdout:(5/30): ceph-radosgw-18.2.0-524.g38c3646d.el9.x 7.6 MB/s | 7.7 MB 00:01 2023-09-29T22:05:39.049 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:05:39.050 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:05:39.139 INFO:teuthology.orchestra.run.smithi184.stdout:(6/30): librgw2-18.2.0-524.g38c3646d.el9.x86_64 14 MB/s | 4.4 MB 00:00 2023-09-29T22:05:39.164 INFO:teuthology.orchestra.run.smithi184.stdout:(7/30): python3-ceph-argparse-18.2.0-524.g38c36 386 kB/s | 45 kB 00:00 2023-09-29T22:05:39.189 INFO:teuthology.orchestra.run.smithi184.stdout:(8/30): python3-ceph-common-18.2.0-524.g38c3646 2.4 MB/s | 124 kB 00:00 2023-09-29T22:05:39.215 INFO:teuthology.orchestra.run.smithi184.stdout:(9/30): python3-cephfs-18.2.0-524.g38c3646d.el9 3.0 MB/s | 156 kB 00:00 2023-09-29T22:05:39.240 INFO:teuthology.orchestra.run.smithi184.stdout:(10/30): python3-rados-18.2.0-524.g38c3646d.el9 6.3 MB/s | 321 kB 00:00 2023-09-29T22:05:39.265 INFO:teuthology.orchestra.run.smithi184.stdout:(11/30): python3-rbd-18.2.0-524.g38c3646d.el9.x 5.8 MB/s | 298 kB 00:00 2023-09-29T22:05:39.290 INFO:teuthology.orchestra.run.smithi184.stdout:(12/30): python3-rgw-18.2.0-524.g38c3646d.el9.x 1.9 MB/s | 99 kB 00:00 2023-09-29T22:05:39.408 INFO:teuthology.orchestra.run.smithi184.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2023-09-29T22:05:39.450 INFO:teuthology.orchestra.run.smithi184.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2023-09-29T22:05:39.491 INFO:teuthology.orchestra.run.smithi184.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2023-09-29T22:05:39.517 INFO:teuthology.orchestra.run.smithi184.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-09-29T22:05:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2023-09-29T22:05:39.634 INFO:teuthology.orchestra.run.smithi184.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2023-09-29T22:05:40.168 INFO:teuthology.orchestra.run.smithi184.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 175 kB/s | 111 kB 00:00 2023-09-29T22:05:40.310 INFO:teuthology.orchestra.run.smithi184.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 457 kB/s | 309 kB 00:00 2023-09-29T22:05:40.415 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:05:40.469 INFO:teuthology.orchestra.run.smithi184.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 154 kB/s | 24 kB 00:00 2023-09-29T22:05:40.481 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-09-29T22:05:40.532 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-09-29T22:05:40.590 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-09-29T22:05:40.794 INFO:teuthology.orchestra.run.smithi184.stdout:(22/30): ceph-common-18.2.0-524.g38c3646d.el9.x 6.6 MB/s | 18 MB 00:02 2023-09-29T22:05:40.819 INFO:teuthology.orchestra.run.smithi184.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2023-09-29T22:05:40.954 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-09-29T22:05:40.989 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:05:41.080 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:05:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 6/32 2023-09-29T22:05:41.182 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:05:41.219 INFO:teuthology.orchestra.run.smithi184.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 158 kB/s | 67 kB 00:00 2023-09-29T22:05:41.228 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:05:41.253 INFO:teuthology.orchestra.run.smithi184.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.9 MB/s | 837 kB 00:00 2023-09-29T22:05:41.256 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-09-29T22:05:41.299 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x 9/32 2023-09-29T22:05:41.360 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 10/32 2023-09-29T22:05:41.384 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:05:41.475 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:05:41.536 INFO:teuthology.orchestra.run.smithi184.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 604 kB/s | 192 kB 00:00 2023-09-29T22:05:41.580 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-09-29T22:05:41.617 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-09-29T22:05:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-09-29T22:05:41.687 INFO:teuthology.orchestra.run.smithi184.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2023-09-29T22:05:41.698 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-09-29T22:05:41.725 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-09-29T22:05:41.753 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-09-29T22:05:41.786 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-29T22:05:41.804 INFO:teuthology.orchestra.run.smithi184.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2023-09-29T22:05:41.879 INFO:teuthology.orchestra.run.smithi184.stdout:(29/30): librados2-18.2.0-524.g38c3646d.el9.x86 9.6 MB/s | 3.3 MB 00:00 2023-09-29T22:05:42.071 INFO:teuthology.orchestra.run.smithi184.stdout:(30/30): librbd1-18.2.0-524.g38c3646d.el9.x86_6 7.8 MB/s | 3.0 MB 00:00 2023-09-29T22:05:42.074 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:05:42.074 INFO:teuthology.orchestra.run.smithi184.stdout:Total 12 MB/s | 53 MB 00:04 2023-09-29T22:05:42.084 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-09-29T22:05:42.087 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:05:42.116 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 20/32 2023-09-29T22:05:42.123 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:05:42.124 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:05:42.189 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 20/32 2023-09-29T22:05:42.220 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 21/32 2023-09-29T22:05:42.264 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-09-29T22:05:42.648 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:05:42.649 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:05:42.654 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-09-29T22:05:42.685 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 24/32 2023-09-29T22:05:42.747 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 24/32 2023-09-29T22:05:42.814 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 25/32 2023-09-29T22:05:42.845 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86 26/32 2023-09-29T22:05:44.013 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:05:44.075 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-09-29T22:05:44.117 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-09-29T22:05:44.184 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-09-29T22:05:44.554 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-09-29T22:05:44.599 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:05:44.674 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:05:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 6/32 2023-09-29T22:05:44.768 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:05:44.814 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:05:44.842 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-09-29T22:05:44.885 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x 9/32 2023-09-29T22:05:44.958 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 10/32 2023-09-29T22:05:44.986 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:05:45.044 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:05:45.108 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:45.149 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-09-29T22:05:45.151 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:45.186 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-09-29T22:05:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-09-29T22:05:45.267 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-09-29T22:05:45.294 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-09-29T22:05:45.330 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-09-29T22:05:45.362 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-29T22:05:45.661 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-09-29T22:05:45.684 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 20/32 2023-09-29T22:05:45.692 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:45.746 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 28/32 2023-09-29T22:05:45.757 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 20/32 2023-09-29T22:05:45.789 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 21/32 2023-09-29T22:05:45.825 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-09-29T22:05:45.829 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 28/32 2023-09-29T22:05:45.829 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-09-29T22:05:45.829 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-09-29T22:05:45.829 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:45.872 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:05:46.225 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-09-29T22:05:46.254 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 24/32 2023-09-29T22:05:46.308 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 24/32 2023-09-29T22:05:46.375 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 25/32 2023-09-29T22:05:46.406 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86 26/32 2023-09-29T22:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:48.703 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:49.260 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 27/32 2023-09-29T22:05:49.306 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 28/32 2023-09-29T22:05:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 28/32 2023-09-29T22:05:49.385 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-09-29T22:05:49.385 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-09-29T22:05:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:05:49.416 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:05:55.417 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /root 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2023-09-29T22:05:55.418 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 30/32 2023-09-29T22:05:56.196 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 30/32 2023-09-29T22:05:56.196 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:05:56.196 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-09-29T22:05:56.197 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:05:56.197 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:05:56.197 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:56.198 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-29T22:05:56.283 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-29T22:05:56.283 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:05:57.361 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:05:57.361 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 1/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 3/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 4/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 6/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x 8/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86 9/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 10/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:05:57.362 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 12/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 13/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-09-29T22:05:57.363 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-09-29T22:05:57.364 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 31/32 2023-09-29T22:05:58.509 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.510 INFO:teuthology.orchestra.run.smithi175.stdout: fmt-8.1.1-5.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-9.0.0-9.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.7-2.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.6.2-1.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-09-29T22:05:58.511 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: re2-1:20211101-3.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.15.0-2.el9.x86_64 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:58.512 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:05:58.866 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-test 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2023-09-29T22:05:58.902 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2023-09-29T22:05:58.903 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /root 2023-09-29T22:05:58.903 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2023-09-29T22:05:58.903 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:05:59.433 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:05:59.579 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:05:59.580 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:59.580 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:05:59.580 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:59.581 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:05:59.581 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:18.2.0-524.g38c3646d.el9 ceph 40 M 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:18.2.0-524.g38c3646d.el9 ceph 161 k 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-09-29T22:05:59.582 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:59.583 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:05:59.583 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:05:59.583 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2023-09-29T22:05:59.583 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:05:59.584 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 40 M 2023-09-29T22:05:59.584 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 167 M 2023-09-29T22:05:59.584 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:05:59.659 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 30/32 2023-09-29T22:05:59.689 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 30/32 2023-09-29T22:05:59.689 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:05:59.689 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-09-29T22:05:59.689 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:05:59.689 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-29T22:05:59.690 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:05:59.691 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-29T22:05:59.761 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-29T22:05:59.762 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:05:59.858 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-09-29T22:05:59.908 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2023-09-29T22:05:59.942 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-09-29T22:06:00.084 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): libcephsqlite-18.2.0-524.g38c3646d.el9.x 428 kB/s | 161 kB 00:00 2023-09-29T22:06:00.167 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2023-09-29T22:06:00.848 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:06:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 1/32 2023-09-29T22:06:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2/32 2023-09-29T22:06:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 3/32 2023-09-29T22:06:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 4/32 2023-09-29T22:06:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 5/32 2023-09-29T22:06:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 6/32 2023-09-29T22:06:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 7/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x 8/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86 9/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 10/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 11/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 12/32 2023-09-29T22:06:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 13/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-09-29T22:06:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-09-29T22:06:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-09-29T22:06:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-09-29T22:06:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:18.2.0-524.g38c3646d.el9.x86_64 29/32 2023-09-29T22:06:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-09-29T22:06:01.032 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 31/32 2023-09-29T22:06:01.859 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): ceph-test-18.2.0-524.g38c3646d.el9.x86_6 18 MB/s | 40 MB 00:02 2023-09-29T22:06:01.862 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:01.862 INFO:teuthology.orchestra.run.smithi175.stdout:Total 18 MB/s | 40 MB 00:02 2023-09-29T22:06:01.876 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:01.897 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:01.898 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: fmt-8.1.1-5.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-9.0.0-9.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.7-2.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-09-29T22:06:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.6.2-1.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: re2-1:20211101-3.el9.x86_64 2023-09-29T22:06:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: thrift-0.15.0-2.el9.x86_64 2023-09-29T22:06:01.908 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:01.908 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:02.234 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:02.234 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:02.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2023-09-29T22:06:02.762 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:02.820 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:02.834 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-09-29T22:06:02.868 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-09-29T22:06:02.900 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-09-29T22:06:02.942 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-09-29T22:06:02.967 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:18.2.0-524.g38c3646d.el9 ceph 40 M 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:02.968 INFO:teuthology.orchestra.run.smithi184.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:18.2.0-524.g38c3646d.el9 ceph 161 k 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2023-09-29T22:06:02.969 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:02.970 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 40 M 2023-09-29T22:06:02.970 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 167 M 2023-09-29T22:06:02.970 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:02.971 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 5/6 2023-09-29T22:06:03.386 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-09-29T22:06:03.437 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2023-09-29T22:06:03.470 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-09-29T22:06:03.579 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): libcephsqlite-18.2.0-524.g38c3646d.el9.x 464 kB/s | 161 kB 00:00 2023-09-29T22:06:04.012 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 345 kB/s | 187 kB 00:00 2023-09-29T22:06:05.787 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): ceph-test-18.2.0-524.g38c3646d.el9.x86_6 15 MB/s | 40 MB 00:02 2023-09-29T22:06:05.790 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:05.791 INFO:teuthology.orchestra.run.smithi184.stdout:Total 14 MB/s | 40 MB 00:02 2023-09-29T22:06:05.804 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:05.826 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:05.826 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:06.166 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:06.167 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:06.696 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:06.698 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 5/6 2023-09-29T22:06:06.768 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 6/6 2023-09-29T22:06:06.779 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-09-29T22:06:06.813 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-09-29T22:06:06.845 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-09-29T22:06:06.887 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-09-29T22:06:06.915 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 5/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 6/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 1/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 2/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-09-29T22:06:08.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-09-29T22:06:09.162 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-09-29T22:06:09.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: jq-1.6-15.el9.x86_64 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.4.1-5.el9.x86_64 2023-09-29T22:06:09.163 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-09-29T22:06:09.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:09.164 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:09.413 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph 2023-09-29T22:06:09.992 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:37 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:10.138 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:10.140 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2023-09-29T22:06:10.140 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:18.2.0-524.g38c3646d.el9 ceph 6.5 k 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:18.2.0-524.g38c3646d.el9 ceph 2.1 M 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:18.2.0-524.g38c3646d.el9 ceph 1.4 M 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 241 k 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:18.2.0-524.g38c3646d.el9 ceph 4.4 M 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 18 M 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-09-29T22:06:10.141 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-09-29T22:06:10.142 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-09-29T22:06:10.143 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout:Install 33 Packages 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:10.144 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 30 M 2023-09-29T22:06:10.145 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 109 M 2023-09-29T22:06:10.145 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:10.670 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 5/6 2023-09-29T22:06:10.730 INFO:teuthology.orchestra.run.smithi175.stdout:(1/33): ceph-18.2.0-524.g38c3646d.el9.x86_64.rp 29 kB/s | 6.5 kB 00:00 2023-09-29T22:06:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 6/6 2023-09-29T22:06:10.822 INFO:teuthology.orchestra.run.smithi175.stdout:(2/33): ceph-mgr-18.2.0-524.g38c3646d.el9.x86_6 4.5 MB/s | 1.4 MB 00:00 2023-09-29T22:06:10.872 INFO:teuthology.orchestra.run.smithi175.stdout:(3/33): ceph-mds-18.2.0-524.g38c3646d.el9.x86_6 5.7 MB/s | 2.1 MB 00:00 2023-09-29T22:06:10.981 INFO:teuthology.orchestra.run.smithi175.stdout:(4/33): ceph-mgr-modules-core-18.2.0-524.g38c36 2.2 MB/s | 241 kB 00:00 2023-09-29T22:06:11.081 INFO:teuthology.orchestra.run.smithi175.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-09-29T22:06:11.114 INFO:teuthology.orchestra.run.smithi175.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-09-29T22:06:11.148 INFO:teuthology.orchestra.run.smithi175.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-09-29T22:06:11.190 INFO:teuthology.orchestra.run.smithi175.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-09-29T22:06:11.223 INFO:teuthology.orchestra.run.smithi175.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-09-29T22:06:11.257 INFO:teuthology.orchestra.run.smithi175.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-09-29T22:06:11.291 INFO:teuthology.orchestra.run.smithi175.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-09-29T22:06:11.333 INFO:teuthology.orchestra.run.smithi175.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-09-29T22:06:11.675 INFO:teuthology.orchestra.run.smithi175.stdout:(13/33): ceph-osd-18.2.0-524.g38c3646d.el9.x86_ 21 MB/s | 18 MB 00:00 2023-09-29T22:06:11.950 INFO:teuthology.orchestra.run.smithi175.stdout:(14/33): ceph-mon-18.2.0-524.g38c3646d.el9.x86_ 3.6 MB/s | 4.4 MB 00:01 2023-09-29T22:06:12.083 INFO:teuthology.orchestra.run.smithi175.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2023-09-29T22:06:12.117 INFO:teuthology.orchestra.run.smithi175.stdout:(16/33): python3-cryptography-36.0.1-4.el9.x86_ 1.6 MB/s | 1.2 MB 00:00 2023-09-29T22:06:12.176 INFO:teuthology.orchestra.run.smithi175.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2023-09-29T22:06:12.226 INFO:teuthology.orchestra.run.smithi175.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2023-09-29T22:06:12.268 INFO:teuthology.orchestra.run.smithi175.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2023-09-29T22:06:12.293 INFO:teuthology.orchestra.run.smithi175.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2023-09-29T22:06:12.335 INFO:teuthology.orchestra.run.smithi175.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2023-09-29T22:06:12.360 INFO:teuthology.orchestra.run.smithi175.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2023-09-29T22:06:12.385 INFO:teuthology.orchestra.run.smithi175.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2023-09-29T22:06:12.411 INFO:teuthology.orchestra.run.smithi175.stdout:(24/33): python3-pytz-2021.1-5.el9.noarch.rpm 69 kB/s | 51 kB 00:00 2023-09-29T22:06:12.436 INFO:teuthology.orchestra.run.smithi175.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2023-09-29T22:06:12.461 INFO:teuthology.orchestra.run.smithi175.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-09-29T22:06:12.503 INFO:teuthology.orchestra.run.smithi175.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2023-09-29T22:06:12.520 INFO:teuthology.orchestra.run.smithi175.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-09-29T22:06:12.554 INFO:teuthology.orchestra.run.smithi175.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2023-09-29T22:06:12.579 INFO:teuthology.orchestra.run.smithi175.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2023-09-29T22:06:12.605 INFO:teuthology.orchestra.run.smithi175.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-09-29T22:06:12.630 INFO:teuthology.orchestra.run.smithi175.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2023-09-29T22:06:12.654 INFO:teuthology.orchestra.run.smithi175.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2023-09-29T22:06:12.658 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:12.659 INFO:teuthology.orchestra.run.smithi175.stdout:Total 12 MB/s | 30 MB 00:02 2023-09-29T22:06:12.692 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:12.731 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:12.732 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 6/6 2023-09-29T22:06:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 1/6 2023-09-29T22:06:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 2/6 2023-09-29T22:06:12.797 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-09-29T22:06:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-09-29T22:06:12.798 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-09-29T22:06:13.092 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:13.093 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:13.199 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-09-29T22:06:13.199 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:13.199 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:13.199 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: jq-1.6-15.el9.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-5.el9.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:13.200 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:13.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2023-09-29T22:06:14.059 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:39 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:14.202 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:18.2.0-524.g38c3646d.el9 ceph 6.5 k 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:18.2.0-524.g38c3646d.el9 ceph 2.1 M 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:18.2.0-524.g38c3646d.el9 ceph 1.4 M 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 241 k 2023-09-29T22:06:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:18.2.0-524.g38c3646d.el9 ceph 4.4 M 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 18 M 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-09-29T22:06:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-09-29T22:06:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================== 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout:Install 33 Packages 2023-09-29T22:06:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 30 M 2023-09-29T22:06:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 109 M 2023-09-29T22:06:14.209 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:14.319 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:14.374 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-09-29T22:06:14.406 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-09-29T22:06:14.456 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-09-29T22:06:14.547 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-09-29T22:06:14.625 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-09-29T22:06:14.666 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-09-29T22:06:14.712 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-09-29T22:06:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-09-29T22:06:14.834 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-09-29T22:06:14.872 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-09-29T22:06:14.873 INFO:teuthology.orchestra.run.smithi184.stdout:(1/33): ceph-18.2.0-524.g38c3646d.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2023-09-29T22:06:14.921 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-09-29T22:06:14.976 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-29T22:06:15.072 INFO:teuthology.orchestra.run.smithi184.stdout:(2/33): ceph-mgr-18.2.0-524.g38c3646d.el9.x86_6 4.0 MB/s | 1.4 MB 00:00 2023-09-29T22:06:15.165 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-09-29T22:06:15.210 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-09-29T22:06:15.241 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-09-29T22:06:15.275 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-09-29T22:06:15.305 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-09-29T22:06:15.364 INFO:teuthology.orchestra.run.smithi184.stdout:(3/33): ceph-mon-18.2.0-524.g38c3646d.el9.x86_6 9.0 MB/s | 4.4 MB 00:00 2023-09-29T22:06:15.380 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-09-29T22:06:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-09-29T22:06:15.423 INFO:teuthology.orchestra.run.smithi184.stdout:(4/33): ceph-mgr-modules-core-18.2.0-524.g38c36 4.0 MB/s | 241 kB 00:00 2023-09-29T22:06:15.449 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-09-29T22:06:15.481 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-09-29T22:06:15.508 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-09-29T22:06:15.523 INFO:teuthology.orchestra.run.smithi184.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-09-29T22:06:15.548 INFO:teuthology.orchestra.run.smithi184.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-09-29T22:06:15.574 INFO:teuthology.orchestra.run.smithi184.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-09-29T22:06:15.596 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-09-29T22:06:15.616 INFO:teuthology.orchestra.run.smithi184.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-09-29T22:06:15.649 INFO:teuthology.orchestra.run.smithi184.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-09-29T22:06:15.650 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-09-29T22:06:15.683 INFO:teuthology.orchestra.run.smithi184.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-09-29T22:06:15.725 INFO:teuthology.orchestra.run.smithi184.stdout:(11/33): ceph-mds-18.2.0-524.g38c3646d.el9.x86_ 2.1 MB/s | 2.1 MB 00:01 2023-09-29T22:06:15.726 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-09-29T22:06:15.750 INFO:teuthology.orchestra.run.smithi184.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 1.6 MB/s | 113 kB 00:00 2023-09-29T22:06:15.792 INFO:teuthology.orchestra.run.smithi184.stdout:(13/33): python3-urllib3-1.26.5-3.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2023-09-29T22:06:15.862 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-09-29T22:06:15.892 INFO:teuthology.orchestra.run.smithi184.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 511 kB/s | 51 kB 00:00 2023-09-29T22:06:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-29T22:06:16.134 INFO:teuthology.orchestra.run.smithi184.stdout:(15/33): ceph-osd-18.2.0-524.g38c3646d.el9.x86_ 17 MB/s | 18 MB 00:01 2023-09-29T22:06:16.142 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.n 28/33 2023-09-29T22:06:16.168 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 29/33 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 29/33 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-29T22:06:16.195 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:16.634 INFO:teuthology.orchestra.run.smithi184.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 59 kB/s | 43 kB 00:00 2023-09-29T22:06:16.818 INFO:teuthology.orchestra.run.smithi184.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 270 kB/s | 172 kB 00:00 2023-09-29T22:06:16.935 INFO:teuthology.orchestra.run.smithi184.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2023-09-29T22:06:16.977 INFO:teuthology.orchestra.run.smithi184.stdout:(19/33): python3-cryptography-36.0.1-4.el9.x86_ 1.0 MB/s | 1.2 MB 00:01 2023-09-29T22:06:17.044 INFO:teuthology.orchestra.run.smithi184.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2023-09-29T22:06:17.069 INFO:teuthology.orchestra.run.smithi184.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 825 kB/s | 358 kB 00:00 2023-09-29T22:06:17.153 INFO:teuthology.orchestra.run.smithi184.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2023-09-29T22:06:17.203 INFO:teuthology.orchestra.run.smithi184.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 179 kB/s | 20 kB 00:00 2023-09-29T22:06:17.261 INFO:teuthology.orchestra.run.smithi184.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2023-09-29T22:06:17.312 INFO:teuthology.orchestra.run.smithi184.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2023-09-29T22:06:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2023-09-29T22:06:17.445 INFO:teuthology.orchestra.run.smithi184.stdout:(27/33): python3-jaraco-collections-3.0.0-8.el9 49 kB/s | 23 kB 00:00 2023-09-29T22:06:17.596 INFO:teuthology.orchestra.run.smithi184.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 489 kB/s | 90 kB 00:00 2023-09-29T22:06:17.646 INFO:teuthology.orchestra.run.smithi184.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 178 kB/s | 36 kB 00:00 2023-09-29T22:06:17.688 INFO:teuthology.orchestra.run.smithi184.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 638 kB/s | 272 kB 00:00 2023-09-29T22:06:17.694 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 30/33 2023-09-29T22:06:17.720 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 30/33 2023-09-29T22:06:17.721 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:17.721 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-09-29T22:06:17.721 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-29T22:06:17.721 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-29T22:06:17.721 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:17.797 INFO:teuthology.orchestra.run.smithi184.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2023-09-29T22:06:17.947 INFO:teuthology.orchestra.run.smithi184.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 654 kB/s | 230 kB 00:00 2023-09-29T22:06:18.178 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 31/33 2023-09-29T22:06:18.201 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 31/33 2023-09-29T22:06:18.202 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:18.202 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-09-29T22:06:18.202 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-29T22:06:18.202 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-29T22:06:18.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:18.231 INFO:teuthology.orchestra.run.smithi184.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 731 kB/s | 427 kB 00:00 2023-09-29T22:06:18.237 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:18.237 INFO:teuthology.orchestra.run.smithi184.stdout:Total 7.4 MB/s | 30 MB 00:04 2023-09-29T22:06:18.269 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:18.309 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:18.309 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:18.404 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 32/33 2023-09-29T22:06:18.427 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 32/33 2023-09-29T22:06:18.427 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:18.427 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-09-29T22:06:18.427 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-29T22:06:18.427 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-29T22:06:18.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:18.493 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:18.2.0-524.g38c3646d.el9.x86_64 33/33 2023-09-29T22:06:18.670 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:18.671 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:19.866 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:19.917 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-09-29T22:06:19.949 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-09-29T22:06:19.981 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-09-29T22:06:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-09-29T22:06:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-09-29T22:06:20.183 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-2:18.2.0-524.g38c3646d.el9.x86_64 33/33 2023-09-29T22:06:20.183 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:18.2.0-524.g38c3646d.el9.x86_64 1/33 2023-09-29T22:06:20.183 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 2/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 3/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 4/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 5/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.n 6/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-09-29T22:06:20.184 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-09-29T22:06:20.185 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-09-29T22:06:20.186 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-09-29T22:06:20.209 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-09-29T22:06:20.255 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-09-29T22:06:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-09-29T22:06:20.385 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-09-29T22:06:20.423 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-09-29T22:06:20.472 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-09-29T22:06:20.528 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-29T22:06:20.715 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-09-29T22:06:20.769 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-09-29T22:06:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-09-29T22:06:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-09-29T22:06:20.865 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-09-29T22:06:20.940 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-09-29T22:06:20.975 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-09-29T22:06:21.008 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-09-29T22:06:21.049 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-09-29T22:06:21.084 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-09-29T22:06:21.125 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-09-29T22:06:21.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.1.4-6.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply-3.11-14.el9.noarch 2023-09-29T22:06:21.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-3.1.0-2.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.20-6.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2021.1-5.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-2.25.1-7.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml-0.10.2-6.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.7-6.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:21.128 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:21.180 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-09-29T22:06:21.226 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-09-29T22:06:21.304 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-09-29T22:06:21.438 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-09-29T22:06:21.446 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-base 2023-09-29T22:06:21.557 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-29T22:06:21.711 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.n 28/33 2023-09-29T22:06:21.745 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 29/33 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 29/33 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-29T22:06:21.770 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:22.051 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:50 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:22.138 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:06:22.186 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:22.187 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:06:22.187 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:22.245 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install cephadm 2023-09-29T22:06:22.821 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:50 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:22.956 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:22.957 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:22.957 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:22.957 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:22.958 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:22.958 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 218 k 2023-09-29T22:06:22.958 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:22.958 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:22.958 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:22.959 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:06:22.959 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:22.959 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 218 k 2023-09-29T22:06:22.959 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 210 k 2023-09-29T22:06:22.959 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:23.271 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 30/33 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 30/33 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-29T22:06:23.297 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:23.300 INFO:teuthology.orchestra.run.smithi175.stdout:cephadm-18.2.0-524.g38c3646d.el9.noarch.rpm 637 kB/s | 218 kB 00:00 2023-09-29T22:06:23.301 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:23.301 INFO:teuthology.orchestra.run.smithi175.stdout:Total 634 kB/s | 218 kB 00:00 2023-09-29T22:06:23.301 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:23.307 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:23.307 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:23.309 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:23.309 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:23.381 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 31/33 2023-09-29T22:06:23.786 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 31/33 2023-09-29T22:06:23.786 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:23.786 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-09-29T22:06:23.787 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-29T22:06:23.787 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-29T22:06:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:23.836 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:23.929 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:23.989 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 32/33 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 32/33 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-29T22:06:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:24.077 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:18.2.0-524.g38c3646d.el9.x86_64 33/33 2023-09-29T22:06:24.329 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:24.626 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:24.833 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T22:06:25.412 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:25.551 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-524.g38c3646d.el9 ceph 142 k 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:25.552 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:06:25.553 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:25.553 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 142 k 2023-09-29T22:06:25.553 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 399 k 2023-09-29T22:06:25.553 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:18.2.0-524.g38c3646d.el9.x86_64 33/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:18.2.0-524.g38c3646d.el9.x86_64 1/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 2/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 3/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 4/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 5/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.n 6/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-09-29T22:06:25.853 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-09-29T22:06:25.854 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-09-29T22:06:25.855 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-09-29T22:06:25.897 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-immutable-object-cache-18.2.0-524.g38c3646 413 kB/s | 142 kB 00:00 2023-09-29T22:06:25.898 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:25.898 INFO:teuthology.orchestra.run.smithi175.stdout:Total 411 kB/s | 142 kB 00:00 2023-09-29T22:06:25.898 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:25.905 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:25.906 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:25.956 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:25.957 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:26.096 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:26.157 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:26.180 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:26.180 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:26.180 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-09-29T22:06:26.180 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-09-29T22:06:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:26.743 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-09-29T22:06:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.1.4-6.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ply-3.11-14.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-3.1.0-2.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-09-29T22:06:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.20-6.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2021.1-5.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-2.25.1-7.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml-0.10.2-6.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.7-6.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:26.746 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:27.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2023-09-29T22:06:27.165 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:27.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:27.165 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:27.166 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:27.166 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:27.166 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:27.399 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr 2023-09-29T22:06:27.643 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:52 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:27.730 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:06:27.777 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:27.778 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:06:27.779 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:27.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2023-09-29T22:06:27.982 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:28.072 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:06:28.120 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:28.121 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:06:28.121 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:28.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T22:06:28.416 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:28.550 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 218 k 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:28.551 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:28.552 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:06:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:28.552 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 218 k 2023-09-29T22:06:28.552 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 210 k 2023-09-29T22:06:28.552 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:28.751 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:28.787 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-18.2.0-524.g38c3646d.el9.noarch.rpm 929 kB/s | 218 kB 00:00 2023-09-29T22:06:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Total 924 kB/s | 218 kB 00:00 2023-09-29T22:06:28.788 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:28.793 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:28.793 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:28.795 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:28.796 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:28.869 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:28.889 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 1.7 M 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 23 k 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 15 k 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-09-29T22:06:28.890 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout:Installing weak dependencies: 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout:Install 7 Packages 2023-09-29T22:06:28.891 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:28.892 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 2.0 M 2023-09-29T22:06:28.892 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 8.4 M 2023-09-29T22:06:28.892 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:29.159 INFO:teuthology.orchestra.run.smithi175.stdout:(1/7): ceph-grafana-dashboards-18.2.0-524.g38c3 115 kB/s | 23 kB 00:00 2023-09-29T22:06:29.188 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:29.200 INFO:teuthology.orchestra.run.smithi175.stdout:(2/7): ceph-prometheus-alerts-18.2.0-524.g38c36 60 kB/s | 15 kB 00:00 2023-09-29T22:06:29.267 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:29.326 INFO:teuthology.orchestra.run.smithi175.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 54 kB/s | 9.0 kB 00:00 2023-09-29T22:06:29.359 INFO:teuthology.orchestra.run.smithi175.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 258 kB/s | 41 kB 00:00 2023-09-29T22:06:29.385 INFO:teuthology.orchestra.run.smithi175.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 526 kB/s | 31 kB 00:00 2023-09-29T22:06:29.435 INFO:teuthology.orchestra.run.smithi175.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2023-09-29T22:06:29.501 INFO:teuthology.orchestra.run.smithi175.stdout:(7/7): ceph-mgr-dashboard-18.2.0-524.g38c3646d. 3.2 MB/s | 1.7 MB 00:00 2023-09-29T22:06:29.505 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:29.505 INFO:teuthology.orchestra.run.smithi175.stdout:Total 3.3 MB/s | 2.0 MB 00:00 2023-09-29T22:06:29.505 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:29.513 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:29.513 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:29.575 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:29.576 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:29.612 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:29.877 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:29.937 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-09-29T22:06:29.979 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-09-29T22:06:30.033 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-09-29T22:06:30.065 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-09-29T22:06:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.no 5/7 2023-09-29T22:06:30.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T22:06:30.326 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.n 6/7 2023-09-29T22:06:30.380 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 7/7 2023-09-29T22:06:30.716 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:30.856 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-524.g38c3646d.el9 ceph 142 k 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:06:30.857 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:30.858 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 142 k 2023-09-29T22:06:30.858 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 399 k 2023-09-29T22:06:30.858 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:30.884 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 7/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.n 1/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 2/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.no 3/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-09-29T22:06:30.885 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-09-29T22:06:31.058 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-18.2.0-524.g38c3646 710 kB/s | 142 kB 00:00 2023-09-29T22:06:31.059 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:31.059 INFO:teuthology.orchestra.run.smithi184.stdout:Total 705 kB/s | 142 kB 00:00 2023-09-29T22:06:31.059 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:31.118 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:31.119 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:31.201 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-09-29T22:06:31.202 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-09-29T22:06:31.202 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-09-29T22:06:31.202 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.5.1-5.el9.noarch 2023-09-29T22:06:31.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:31.202 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:31.253 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:31.318 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:31.342 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:31.342 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-09-29T22:06:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:31.424 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T22:06:31.997 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:32.137 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:32.138 INFO:teuthology.orchestra.run.smithi175.stdout:=========================================================================================== 2023-09-29T22:06:32.138 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:32.138 INFO:teuthology.orchestra.run.smithi175.stdout:=========================================================================================== 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 7.4 M 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 11.4.1-2.1.el9 baseos 809 k 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 11.4.1-2.1.el9 baseos 198 k 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-09-29T22:06:32.139 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout:=========================================================================================== 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout:Install 12 Packages 2023-09-29T22:06:32.140 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:32.141 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 37 M 2023-09-29T22:06:32.141 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 209 M 2023-09-29T22:06:32.141 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:32.287 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.e 1/1 2023-09-29T22:06:32.287 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:32.287 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:32.287 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:32.288 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:32.288 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:32.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2023-09-29T22:06:32.541 INFO:teuthology.orchestra.run.smithi175.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2023-09-29T22:06:32.575 INFO:teuthology.orchestra.run.smithi175.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 529 kB/s | 17 kB 00:00 2023-09-29T22:06:32.617 INFO:teuthology.orchestra.run.smithi175.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 835 kB/s | 34 kB 00:00 2023-09-29T22:06:32.675 INFO:teuthology.orchestra.run.smithi175.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-09-29T22:06:32.767 INFO:teuthology.orchestra.run.smithi175.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 205 kB 00:00 2023-09-29T22:06:32.826 INFO:teuthology.orchestra.run.smithi175.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-09-29T22:06:32.918 INFO:teuthology.orchestra.run.smithi175.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-09-29T22:06:32.943 INFO:teuthology.orchestra.run.smithi175.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-09-29T22:06:33.089 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:33.177 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:06:33.225 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:33.226 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:06:33.226 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:33.276 INFO:teuthology.orchestra.run.smithi175.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2023-09-29T22:06:33.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T22:06:33.468 INFO:teuthology.orchestra.run.smithi175.stdout:(10/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 1.5 MB/s | 809 kB 00:00 2023-09-29T22:06:33.510 INFO:teuthology.orchestra.run.smithi175.stdout:(11/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 850 kB/s | 198 kB 00:00 2023-09-29T22:06:33.854 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:33.990 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:33.990 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 1.7 M 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 23 k 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 15 k 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-09-29T22:06:33.991 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:Install 7 Packages 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:33.992 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.0 M 2023-09-29T22:06:33.993 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 8.4 M 2023-09-29T22:06:33.993 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:34.177 INFO:teuthology.orchestra.run.smithi175.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.0-5 4.1 MB/s | 7.4 MB 00:01 2023-09-29T22:06:34.179 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:34.179 INFO:teuthology.orchestra.run.smithi175.stdout:Total 18 MB/s | 37 MB 00:02 2023-09-29T22:06:34.208 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:34.231 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:34.232 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:34.337 INFO:teuthology.orchestra.run.smithi184.stdout:(1/7): ceph-prometheus-alerts-18.2.0-524.g38c36 58 kB/s | 15 kB 00:00 2023-09-29T22:06:34.370 INFO:teuthology.orchestra.run.smithi184.stdout:(2/7): ceph-grafana-dashboards-18.2.0-524.g38c3 81 kB/s | 23 kB 00:00 2023-09-29T22:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 43 kB/s | 9.0 kB 00:00 2023-09-29T22:06:34.621 INFO:teuthology.orchestra.run.smithi184.stdout:(4/7): ceph-mgr-dashboard-18.2.0-524.g38c3646d. 3.2 MB/s | 1.7 MB 00:00 2023-09-29T22:06:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 153 kB/s | 41 kB 00:00 2023-09-29T22:06:34.643 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:34.643 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:34.763 INFO:teuthology.orchestra.run.smithi184.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 141 kB/s | 31 kB 00:00 2023-09-29T22:06:34.872 INFO:teuthology.orchestra.run.smithi184.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 753 kB/s | 188 kB 00:00 2023-09-29T22:06:34.874 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:34.875 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2023-09-29T22:06:34.875 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:34.883 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:34.883 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:34.951 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:34.951 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:35.242 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:35.290 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-09-29T22:06:35.324 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-09-29T22:06:35.344 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:35.370 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-09-29T22:06:35.402 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-09-29T22:06:35.431 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.no 5/7 2023-09-29T22:06:35.437 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-09-29T22:06:35.473 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-09-29T22:06:35.564 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-09-29T22:06:35.626 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-09-29T22:06:35.664 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.n 6/7 2023-09-29T22:06:35.700 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 7/7 2023-09-29T22:06:35.806 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-29T22:06:35.833 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-29T22:06:35.910 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-09-29T22:06:36.234 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 7/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.n 1/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 2/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.no 3/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-09-29T22:06:36.235 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-09-29T22:06:36.544 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-09-29T22:06:36.544 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:36.544 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:36.544 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.5.1-5.el9.noarch 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:36.545 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:36.571 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-09-29T22:06:36.697 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-09-29T22:06:36.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T22:06:37.352 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:37.489 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout:=========================================================================================== 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout:=========================================================================================== 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 7.4 M 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-09-29T22:06:37.490 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 11.4.1-2.1.el9 baseos 809 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 11.4.1-2.1.el9 baseos 198 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout:=========================================================================================== 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout:Install 12 Packages 2023-09-29T22:06:37.491 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:37.492 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 37 M 2023-09-29T22:06:37.492 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 209 M 2023-09-29T22:06:37.492 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:37.761 INFO:teuthology.orchestra.run.smithi184.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 251 kB/s | 32 kB 00:00 2023-09-29T22:06:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2023-09-29T22:06:37.837 INFO:teuthology.orchestra.run.smithi184.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2023-09-29T22:06:37.895 INFO:teuthology.orchestra.run.smithi184.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-09-29T22:06:37.909 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-09-29T22:06:38.029 INFO:teuthology.orchestra.run.smithi184.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 205 kB 00:00 2023-09-29T22:06:38.096 INFO:teuthology.orchestra.run.smithi184.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-09-29T22:06:38.213 INFO:teuthology.orchestra.run.smithi184.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-09-29T22:06:38.238 INFO:teuthology.orchestra.run.smithi184.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 217 kB 00:00 2023-09-29T22:06:38.614 INFO:teuthology.orchestra.run.smithi184.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2023-09-29T22:06:38.789 INFO:teuthology.orchestra.run.smithi184.stdout:(10/12): ceph-mgr-diskprediction-local-18.2.0-5 6.4 MB/s | 7.4 MB 00:01 2023-09-29T22:06:38.926 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-09-29T22:06:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 12/12 2023-09-29T22:06:39.072 INFO:teuthology.orchestra.run.smithi184.stdout:(11/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 972 kB/s | 809 kB 00:00 2023-09-29T22:06:39.123 INFO:teuthology.orchestra.run.smithi184.stdout:(12/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 390 kB/s | 198 kB 00:00 2023-09-29T22:06:39.125 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:39.125 INFO:teuthology.orchestra.run.smithi184.stdout:Total 23 MB/s | 37 MB 00:01 2023-09-29T22:06:39.154 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:39.178 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:39.178 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:39.598 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:39.599 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:40.302 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:40.389 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-09-29T22:06:40.427 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-09-29T22:06:40.521 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-09-29T22:06:40.563 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-09-29T22:06:40.743 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-29T22:06:40.770 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-29T22:06:40.852 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-09-29T22:06:41.504 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-09-29T22:06:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 12/12 2023-09-29T22:06:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 1/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-09-29T22:06:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 11/12 2023-09-29T22:06:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-09-29T22:06:41.941 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 12/12 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.21-2.el9.x86_64 2023-09-29T22:06:41.942 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:41.943 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:42.280 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-rook 2023-09-29T22:06:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-09-29T22:06:42.865 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:43.009 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:43.010 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:43.010 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:43.010 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 47 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:2.23.0-1.el9 epel 196 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-09-29T22:06:43.011 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout:Install 13 Packages 2023-09-29T22:06:43.012 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:43.013 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 2.0 M 2023-09-29T22:06:43.013 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 26 M 2023-09-29T22:06:43.013 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:43.190 INFO:teuthology.orchestra.run.smithi175.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2023-09-29T22:06:43.215 INFO:teuthology.orchestra.run.smithi175.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-09-29T22:06:43.265 INFO:teuthology.orchestra.run.smithi175.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.5 MB/s | 174 kB 00:00 2023-09-29T22:06:43.290 INFO:teuthology.orchestra.run.smithi175.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-09-29T22:06:43.315 INFO:teuthology.orchestra.run.smithi175.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-09-29T22:06:43.341 INFO:teuthology.orchestra.run.smithi175.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2023-09-29T22:06:43.366 INFO:teuthology.orchestra.run.smithi175.stdout:(7/13): ceph-mgr-rook-18.2.0-524.g38c3646d.el9. 160 kB/s | 47 kB 00:00 2023-09-29T22:06:43.541 INFO:teuthology.orchestra.run.smithi175.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2023-09-29T22:06:43.633 INFO:teuthology.orchestra.run.smithi175.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2023-09-29T22:06:43.767 INFO:teuthology.orchestra.run.smithi175.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2023-09-29T22:06:43.792 INFO:teuthology.orchestra.run.smithi175.stdout:(11/13): python3-google-auth-2.23.0-1.el9.noarc 460 kB/s | 196 kB 00:00 2023-09-29T22:06:43.884 INFO:teuthology.orchestra.run.smithi175.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2023-09-29T22:06:43.891 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-09-29T22:06:43.909 INFO:teuthology.orchestra.run.smithi175.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 631 kB/s | 90 kB 00:00 2023-09-29T22:06:43.913 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:43.913 INFO:teuthology.orchestra.run.smithi175.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2023-09-29T22:06:43.913 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:43.919 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 12/12 2023-09-29T22:06:43.926 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:43.926 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:44.021 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:44.022 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:44.537 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:44.696 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-09-29T22:06:44.751 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-09-29T22:06:44.806 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-09-29T22:06:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-09-29T22:06:44.877 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-09-29T22:06:44.968 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-09-29T22:06:45.078 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:2.23.0-1.el9.noarch 7/13 2023-09-29T22:06:45.128 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-09-29T22:06:45.839 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-09-29T22:06:45.883 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-09-29T22:06:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-09-29T22:06:45.971 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-09-29T22:06:45.996 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 13/13 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 12/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-524.g38c364 1/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-29T22:06:46.491 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-09-29T22:06:46.492 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-09-29T22:06:46.492 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-09-29T22:06:46.492 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-09-29T22:06:46.492 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 11/12 2023-09-29T22:06:46.795 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 13/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 1/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:2.23.0-1.el9.noarch 10/13 2023-09-29T22:06:46.796 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-09-29T22:06:46.797 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-09-29T22:06:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 12/12 2023-09-29T22:06:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:46.948 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.21-2.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-09-29T22:06:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:46.950 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:47.164 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:2.23.0-1.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-09-29T22:06:47.165 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-4.9-2.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:47.166 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:47.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2023-09-29T22:06:47.379 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T22:06:47.881 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:47.954 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:48.019 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 47 k 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-09-29T22:06:48.021 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:2.23.0-1.el9 epel 196 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:48.022 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:48.023 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2023-09-29T22:06:48.023 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:48.023 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 2.0 M 2023-09-29T22:06:48.023 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 26 M 2023-09-29T22:06:48.023 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:48.091 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:48.091 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 132 k 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:48.092 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 6.8 M 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 33 M 2023-09-29T22:06:48.093 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:48.468 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-09-29T22:06:48.477 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 209 kB/s | 24 kB 00:00 2023-09-29T22:06:48.502 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2023-09-29T22:06:48.561 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2023-09-29T22:06:48.577 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-09-29T22:06:48.601 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-09-29T22:06:48.603 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2023-09-29T22:06:48.620 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): ceph-mgr-rook-18.2.0-524.g38c3646d.el9. 183 kB/s | 47 kB 00:00 2023-09-29T22:06:48.637 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 730 kB/s | 43 kB 00:00 2023-09-29T22:06:48.651 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): ceph-mgr-cephadm-18.2.0-524.g38c3646d.el 373 kB/s | 132 kB 00:00 2023-09-29T22:06:48.762 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2023-09-29T22:06:48.779 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2023-09-29T22:06:48.871 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2023-09-29T22:06:48.896 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-google-auth-2.23.0-1.el9.noarc 756 kB/s | 196 kB 00:00 2023-09-29T22:06:48.935 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 186 kB/s | 58 kB 00:00 2023-09-29T22:06:48.960 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2023-09-29T22:06:48.996 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 715 kB/s | 90 kB 00:00 2023-09-29T22:06:49.010 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2023-09-29T22:06:49.014 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:49.015 INFO:teuthology.orchestra.run.smithi175.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2023-09-29T22:06:49.015 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:49.024 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:49.024 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:49.115 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:49.115 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:49.428 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:49.497 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2023-09-29T22:06:49.498 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:49.499 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-09-29T22:06:49.499 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:49.513 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:49.513 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:49.593 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-09-29T22:06:49.611 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:49.612 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:49.654 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-09-29T22:06:50.126 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:50.173 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-09-29T22:06:50.257 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-09-29T22:06:50.301 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-09-29T22:06:50.317 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-09-29T22:06:50.341 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 6/6 2023-09-29T22:06:50.363 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-09-29T22:06:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-09-29T22:06:50.448 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-09-29T22:06:50.489 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-09-29T22:06:50.582 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-09-29T22:06:50.691 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:2.23.0-1.el9.noarch 7/13 2023-09-29T22:06:50.749 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 6/6 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/6 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-09-29T22:06:51.128 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-09-29T22:06:51.431 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.9.1-2.el9.noarch 2023-09-29T22:06:51.432 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-09-29T22:06:51.432 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-09-29T22:06:51.432 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-09-29T22:06:51.432 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:51.432 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:51.468 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-09-29T22:06:51.522 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-09-29T22:06:51.555 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-09-29T22:06:51.634 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-09-29T22:06:51.659 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 13/13 2023-09-29T22:06:51.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-fuse 2023-09-29T22:06:52.259 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:52.395 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 835 k 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:52.396 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 914 k 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 2.7 M 2023-09-29T22:06:52.397 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 13/13 2023-09-29T22:06:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 1/13 2023-09-29T22:06:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-09-29T22:06:52.404 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:2.23.0-1.el9.noarch 10/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-09-29T22:06:52.405 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-09-29T22:06:52.528 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 79 kB 00:00 2023-09-29T22:06:52.788 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:2.23.0-1.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-09-29T22:06:52.789 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el9.noarch 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:52.790 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-fuse-18.2.0-524.g38c3646d.el9.x86_6 1.8 MB/s | 835 kB 00:00 2023-09-29T22:06:52.846 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:52.847 INFO:teuthology.orchestra.run.smithi175.stdout:Total 2.0 MB/s | 914 kB 00:00 2023-09-29T22:06:52.847 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:52.866 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:52.866 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:52.940 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:52.941 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:53.021 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T22:06:53.142 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:53.276 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-09-29T22:06:53.346 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2/2 2023-09-29T22:06:53.597 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:53.732 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 132 k 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:53.733 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout:======================================================================================= 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2023-09-29T22:06:53.734 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:53.735 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 6.8 M 2023-09-29T22:06:53.735 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 33 M 2023-09-29T22:06:53.735 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:53.982 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-09-29T22:06:53.992 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2/2 2023-09-29T22:06:53.992 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/2 2023-09-29T22:06:54.157 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-09-29T22:06:54.282 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): ceph-mgr-cephadm-18.2.0-524.g38c3646d.el 288 kB/s | 132 kB 00:00 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:54.290 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:54.458 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 193 kB/s | 58 kB 00:00 2023-09-29T22:06:54.533 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 997 kB/s | 548 kB 00:00 2023-09-29T22:06:54.566 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2023-09-29T22:06:54.570 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:54.570 INFO:teuthology.orchestra.run.smithi184.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2023-09-29T22:06:54.570 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:54.579 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:54.579 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:54.607 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-volume 2023-09-29T22:06:54.667 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:54.668 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:54.959 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:55.144 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-09-29T22:06:55.193 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:55.201 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-09-29T22:06:55.330 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 262 k 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:55.331 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 262 k 2023-09-29T22:06:55.332 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 1.2 M 2023-09-29T22:06:55.332 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:55.577 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-volume-18.2.0-524.g38c3646d.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2023-09-29T22:06:55.578 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:55.578 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-09-29T22:06:55.578 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:55.586 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:55.586 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:55.619 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:55.619 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:55.732 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-09-29T22:06:55.812 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-09-29T22:06:55.832 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:55.873 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-09-29T22:06:55.897 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 6/6 2023-09-29T22:06:55.901 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:55.916 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:55.917 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:06:55.917 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-09-29T22:06:55.917 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 6/6 2023-09-29T22:06:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 1/6 2023-09-29T22:06:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-09-29T22:06:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-09-29T22:06:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-09-29T22:06:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.9.1-2.el9.noarch 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-09-29T22:06:56.985 INFO:teuthology.orchestra.run.smithi184.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-09-29T22:06:56.986 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:56.986 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:06:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:06:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:57.003 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:06:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:57.003 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:57.221 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2023-09-29T22:06:57.304 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados-devel 2023-09-29T22:06:57.796 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:06:57.886 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:06:57.932 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 835 k 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2023-09-29T22:06:57.933 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 914 k 2023-09-29T22:06:57.934 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 2.7 M 2023-09-29T22:06:57.935 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:06:58.021 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:18.2.0-524.g38c3646d.el9 ceph 126 k 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:06:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:06:58.023 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 126 k 2023-09-29T22:06:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 456 k 2023-09-29T22:06:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:06:58.051 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 676 kB/s | 79 kB 00:00 2023-09-29T22:06:58.251 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): ceph-fuse-18.2.0-524.g38c3646d.el9.x86_6 2.6 MB/s | 835 kB 00:00 2023-09-29T22:06:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.8 MB/s | 914 kB 00:00 2023-09-29T22:06:58.253 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:06:58.273 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:06:58.273 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:06:58.337 INFO:teuthology.orchestra.run.smithi175.stdout:librados-devel-18.2.0-524.g38c3646d.el9.x86_64. 403 kB/s | 126 kB 00:00 2023-09-29T22:06:58.338 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:06:58.338 INFO:teuthology.orchestra.run.smithi175.stdout:Total 401 kB/s | 126 kB 00:00 2023-09-29T22:06:58.338 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:06:58.343 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:06:58.344 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:06:58.348 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:06:58.348 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:06:58.398 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:06:58.398 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:06:58.547 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:06:58.561 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:06:58.643 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:06:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-09-29T22:06:58.759 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2/2 2023-09-29T22:06:59.144 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:06:59.432 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:06:59.485 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2/2 2023-09-29T22:06:59.485 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/2 2023-09-29T22:06:59.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs2 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:06:59.823 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:00.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-volume 2023-09-29T22:07:00.235 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:00.325 INFO:teuthology.orchestra.run.smithi175.stdout:Package libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:00.376 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:00.377 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:00.377 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:00.447 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs-devel 2023-09-29T22:07:00.677 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:00.814 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume noarch 2:18.2.0-524.g38c3646d.el9 ceph-noarch 262 k 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:00.815 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 262 k 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 1.2 M 2023-09-29T22:07:00.816 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:01.010 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:01.041 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-volume-18.2.0-524.g38c3646d.el9.noarch.rpm 1.1 MB/s | 262 kB 00:00 2023-09-29T22:07:01.042 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:01.042 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-09-29T22:07:01.042 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:01.049 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:01.049 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:01.079 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:01.079 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:01.147 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:18.2.0-524.g38c3646d.el9 ceph 31 k 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:07:01.148 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:01.149 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 k 2023-09-29T22:07:01.149 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 137 k 2023-09-29T22:07:01.149 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:07:01.290 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:01.315 INFO:teuthology.orchestra.run.smithi175.stdout:libcephfs-devel-18.2.0-524.g38c3646d.el9.x86_64 184 kB/s | 31 kB 00:00 2023-09-29T22:07:01.316 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:01.316 INFO:teuthology.orchestra.run.smithi175.stdout:Total 183 kB/s | 31 kB 00:00 2023-09-29T22:07:01.316 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:07:01.319 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:07:01.319 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:07:01.346 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:07:01.346 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:07:01.367 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:07:01.382 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:07:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:07:01.382 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-09-29T22:07:01.383 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:01.439 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:07:01.566 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:01.932 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:02.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:02.192 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:02.192 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:07:02.193 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:02.193 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:02.193 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:02.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados2 2023-09-29T22:07:02.497 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 1/1 2023-09-29T22:07:02.497 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:02.497 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:02.497 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-volume-2:18.2.0-524.g38c3646d.el9.noarch 2023-09-29T22:07:02.498 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:02.498 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:02.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2023-09-29T22:07:02.950 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:03.038 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:03.086 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:03.087 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:03.087 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:03.162 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librbd1 2023-09-29T22:07:03.397 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:03.532 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:03.533 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:03.533 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-09-29T22:07:03.533 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:03.533 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:18.2.0-524.g38c3646d.el9 ceph 126 k 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 126 k 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 456 k 2023-09-29T22:07:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:03.736 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:03.745 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-18.2.0-524.g38c3646d.el9.x86_64. 600 kB/s | 126 kB 00:00 2023-09-29T22:07:03.746 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:03.746 INFO:teuthology.orchestra.run.smithi184.stdout:Total 596 kB/s | 126 kB 00:00 2023-09-29T22:07:03.746 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:03.752 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:03.752 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:03.807 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:03.807 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:03.825 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:03.873 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:03.874 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:03.874 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:03.945 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rados 2023-09-29T22:07:03.960 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:04.068 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:04.509 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:04.548 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:04.597 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:04.645 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:04.646 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:04.646 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:04.713 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rgw 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:04.848 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:05.063 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2023-09-29T22:07:05.276 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:05.363 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:05.412 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:05.413 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:05.413 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:05.487 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-cephfs 2023-09-29T22:07:05.651 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:05.743 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:05.793 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:05.794 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:05.795 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:05.864 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2023-09-29T22:07:06.057 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:06.144 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:06.192 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:06.194 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:06.194 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:06.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rbd 2023-09-29T22:07:06.433 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:06.569 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:18.2.0-524.g38c3646d.el9 ceph 31 k 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:06.570 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 k 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 137 k 2023-09-29T22:07:06.571 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:06.815 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-18.2.0-524.g38c3646d.el9.x86_64 125 kB/s | 31 kB 00:00 2023-09-29T22:07:06.816 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:06.816 INFO:teuthology.orchestra.run.smithi184.stdout:Total 125 kB/s | 31 kB 00:00 2023-09-29T22:07:06.816 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:06.819 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:06.820 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:06.825 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:06.857 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:06.857 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:06.913 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:06.944 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:06.963 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:06.964 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2023-09-29T22:07:06.964 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:07.036 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-fuse 2023-09-29T22:07:07.038 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:07.610 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:07.688 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:07.688 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:07.688 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:07.688 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:07.689 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:07.689 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:07.746 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 86 k 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:07.747 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:07:07.748 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:07.748 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 86 k 2023-09-29T22:07:07.748 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 230 k 2023-09-29T22:07:07.748 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:07:07.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2023-09-29T22:07:07.959 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-fuse-18.2.0-524.g38c3646d.el9.x86_64.rpm 407 kB/s | 86 kB 00:00 2023-09-29T22:07:07.960 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:07.960 INFO:teuthology.orchestra.run.smithi175.stdout:Total 405 kB/s | 86 kB 00:00 2023-09-29T22:07:07.960 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:07:07.967 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:07:07.967 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:07:08.021 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:07:08.021 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:07:08.156 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:07:08.248 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:08.443 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:08.531 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:08.580 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:08.581 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:08.581 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:08.638 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:08.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:08.959 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:09.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-mirror 2023-09-29T22:07:09.227 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:09.315 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:09.364 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:09.365 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:09.365 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:09.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2023-09-29T22:07:09.768 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:09.908 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:09.909 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:09.909 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2023-09-29T22:07:09.909 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.0 M 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.0 M 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 12 M 2023-09-29T22:07:09.910 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:07:09.999 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:10.088 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:10.136 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:10.136 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:10.137 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:10.196 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2023-09-29T22:07:10.760 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:10.848 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:10.896 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:10.897 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:10.897 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:10.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2023-09-29T22:07:11.130 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-mirror-18.2.0-524.g38c3646d.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2023-09-29T22:07:11.130 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:11.131 INFO:teuthology.orchestra.run.smithi175.stdout:Total 2.4 MB/s | 3.0 MB 00:01 2023-09-29T22:07:11.131 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:07:11.141 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:07:11.141 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:07:11.213 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:07:11.213 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:07:11.535 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:11.622 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:11.633 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:07:11.671 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:11.672 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:11.672 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:11.697 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-29T22:07:11.721 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:11.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2023-09-29T22:07:12.301 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:12.389 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:18.2.0-524.g38c3646d.el9.x86_64 is already installed. 2023-09-29T22:07:12.438 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:12.439 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2023-09-29T22:07:12.439 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:12.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2023-09-29T22:07:12.868 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:12.868 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:12.869 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:07:12.869 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:12.869 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:12.869 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:13.076 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:13.099 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-nbd 2023-09-29T22:07:13.213 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:13.215 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:13.215 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-09-29T22:07:13.215 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:13.215 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:13.215 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:18.2.0-524.g38c3646d.el9 ceph 86 k 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 86 k 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 230 k 2023-09-29T22:07:13.216 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-18.2.0-524.g38c3646d.el9.x86_64.rpm 463 kB/s | 86 kB 00:00 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi184.stdout:Total 460 kB/s | 86 kB 00:00 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:13.409 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:13.409 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:13.463 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:13.463 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:13.606 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:13.687 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Sep 2023 10:05:32 PM UTC. 2023-09-29T22:07:13.708 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:13.824 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 169 k 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:13.825 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 169 k 2023-09-29T22:07:13.826 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 490 k 2023-09-29T22:07:13.826 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2023-09-29T22:07:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:14.166 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-nbd-18.2.0-524.g38c3646d.el9.x86_64.rpm 497 kB/s | 169 kB 00:00 2023-09-29T22:07:14.167 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:14.167 INFO:teuthology.orchestra.run.smithi175.stdout:Total 495 kB/s | 169 kB 00:00 2023-09-29T22:07:14.167 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2023-09-29T22:07:14.175 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2023-09-29T22:07:14.175 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2023-09-29T22:07:14.230 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2023-09-29T22:07:14.231 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2023-09-29T22:07:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:14.372 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:14.373 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:14.382 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2023-09-29T22:07:14.481 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:14.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2023-09-29T22:07:14.938 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:15.197 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:15.268 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:15.268 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:15.268 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2023-09-29T22:07:15.268 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:15.269 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:15.269 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2023-09-29T22:07:15.338 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:18.2.0-524.g38c3646d.el9 ceph 3.0 M 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:15.339 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:15.340 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:15.340 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:15.340 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.0 M 2023-09-29T22:07:15.340 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 12 M 2023-09-29T22:07:15.340 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:15.523 DEBUG:teuthology.parallel:result is None 2023-09-29T22:07:16.379 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-18.2.0-524.g38c3646d.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2023-09-29T22:07:16.380 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:16.380 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.9 MB/s | 3.0 MB 00:01 2023-09-29T22:07:16.380 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:16.389 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:16.389 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:16.460 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:16.461 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-29T22:07:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:18.104 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:18.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:18.105 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:18.105 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:18.105 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:18.105 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:18.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2023-09-29T22:07:18.926 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Sep 2023 10:05:35 PM UTC. 2023-09-29T22:07:19.063 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2023-09-29T22:07:19.063 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:18.2.0-524.g38c3646d.el9 ceph 169 k 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 169 k 2023-09-29T22:07:19.064 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 490 k 2023-09-29T22:07:19.065 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2023-09-29T22:07:19.265 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-18.2.0-524.g38c3646d.el9.x86_64.rpm 844 kB/s | 169 kB 00:00 2023-09-29T22:07:19.266 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:07:19.266 INFO:teuthology.orchestra.run.smithi184.stdout:Total 839 kB/s | 169 kB 00:00 2023-09-29T22:07:19.266 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2023-09-29T22:07:19.274 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2023-09-29T22:07:19.274 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2023-09-29T22:07:19.330 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2023-09-29T22:07:19.330 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2023-09-29T22:07:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2023-09-29T22:07:19.581 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:19.988 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 1/1 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:18.2.0-524.g38c3646d.el9.x86_64 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:20.269 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2023-09-29T22:07:20.499 DEBUG:teuthology.parallel:result is None 2023-09-29T22:07:20.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T22:07:20.648 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T22:07:20.673 INFO:teuthology.orchestra.run.smithi175.stdout:18.2.0-524.g38c3646d.el9 2023-09-29T22:07:20.674 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-524.g38c3646d.el9 2023-09-29T22:07:20.674 INFO:teuthology.task.install:The correct ceph version 18.2.0-524.g38c3646d is installed. 2023-09-29T22:07:20.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T22:07:20.909 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T22:07:20.930 INFO:teuthology.orchestra.run.smithi184.stdout:18.2.0-524.g38c3646d.el9 2023-09-29T22:07:20.930 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-524.g38c3646d.el9 2023-09-29T22:07:20.930 INFO:teuthology.task.install:The correct ceph version 18.2.0-524.g38c3646d is installed. 2023-09-29T22:07:20.930 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-29T22:07:20.931 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:20.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T22:07:20.964 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:20.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T22:07:20.998 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-29T22:07:20.999 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:20.999 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T22:07:21.035 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T22:07:21.105 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:21.106 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T22:07:21.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T22:07:21.202 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-29T22:07:21.202 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:21.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T22:07:21.231 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T22:07:21.300 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:21.300 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T22:07:21.326 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T22:07:21.393 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-29T22:07:21.393 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:21.393 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T22:07:21.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T22:07:21.492 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:21.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T22:07:21.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T22:07:21.586 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T22:07:21.655 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-29T22:07:21.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /var/log/ceph 2023-09-29T22:07:21.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2023-09-29T22:07:21.685 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-29T22:07:21.685 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T22:07:21.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T22:07:21.754 INFO:tasks.ceph:Creating extra log directories... 2023-09-29T22:07:21.754 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T22:07:21.795 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T22:07:21.831 INFO:tasks.ceph:Creating ceph cluster c1... 2023-09-29T22:07:21.831 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T22:07:21.831 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406303', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4212, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'targets': {'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGg1Rj0/eERk8ONdHWi2h+CdqkStQ5MR5QKLDJW4g0aqKq3EScLjXaZ/6ZMMiGzDXOObRTOT0dqrZkf9wcsjTQk=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjK/BJMklETggsMW2i4eJ/4RHkZRjVcDsj06GMnEus1nm4X8DpnPcTSVd41NffPZrjXOA+ewpgniZENFjXO//c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_12:36:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T22:07:21.832 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-29T22:07:21.863 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-29T22:07:21.877 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T22:07:21.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T22:07:21.945 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:21.945 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T22:07:21.998 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T22:07:21.998 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2023-09-29T22:07:22.052 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T22:07:22.052 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:22.052 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-09-29 22:07:11.969413106 +0000 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-09-29 22:04:49.174625943 +0000 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-09-29 22:04:49.174625943 +0000 2023-09-29T22:07:22.053 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2023-09-29 22:04:49.174625943 +0000 2023-09-29T22:07:22.053 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T22:07:22.118 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-09-29T22:07:22.118 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-09-29T22:07:22.118 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000175877 s, 2.9 MB/s 2023-09-29T22:07:22.119 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T22:07:22.174 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:22.229 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-09-29 22:07:11.970413090 +0000 2023-09-29T22:07:22.230 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-09-29 22:04:49.657618034 +0000 2023-09-29T22:07:22.230 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-09-29 22:04:49.657618034 +0000 2023-09-29T22:07:22.230 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2023-09-29 22:04:49.657618034 +0000 2023-09-29T22:07:22.230 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T22:07:22.295 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-09-29T22:07:22.295 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-09-29T22:07:22.295 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000207301 s, 2.5 MB/s 2023-09-29T22:07:22.296 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T22:07:22.352 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2023-09-29T22:07:22.406 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-09-29 22:07:11.970413090 +0000 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-09-29 22:04:50.162609764 +0000 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-09-29 22:04:50.162609764 +0000 2023-09-29T22:07:22.407 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2023-09-29 22:04:50.162609764 +0000 2023-09-29T22:07:22.408 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T22:07:22.471 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-09-29T22:07:22.472 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-09-29T22:07:22.472 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000209759 s, 2.4 MB/s 2023-09-29T22:07:22.473 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T22:07:22.528 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2023-09-29T22:07:22.582 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T22:07:22.582 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2023-09-29 22:07:11.970413090 +0000 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2023-09-29 22:04:50.636602002 +0000 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2023-09-29 22:04:50.636602002 +0000 2023-09-29T22:07:22.583 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2023-09-29 22:04:50.636602002 +0000 2023-09-29T22:07:22.583 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T22:07:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2023-09-29T22:07:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2023-09-29T22:07:22.648 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000164187 s, 3.1 MB/s 2023-09-29T22:07:22.649 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T22:07:22.704 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:22.704 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:07:22.704 INFO:tasks.ceph:Generating config... 2023-09-29T22:07:22.705 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-29T22:07:22.705 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-29T22:07:22.705 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-29T22:07:22.705 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client] setuser = ceph 2023-09-29T22:07:22.706 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-29T22:07:22.707 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T22:07:22.707 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T22:07:22.707 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T22:07:22.707 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T22:07:22.707 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:22.707 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:22.708 INFO:tasks.ceph:Setting up c1.mon.a... 2023-09-29T22:07:22.708 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-09-29T22:07:22.806 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.keyring 2023-09-29T22:07:22.809 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-09-29T22:07:22.853 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T22:07:22.879 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.175')] 2023-09-29T22:07:22.879 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.175'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-29T22:07:22.880 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:22.880 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:07:22.933 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.175 --print /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:23.010 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:23.010 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:23.011 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:23.011 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: generated fsid 0de4d642-32d1-4a19-89dd-cd5133fa56cc 2023-09-29T22:07:23.011 INFO:teuthology.orchestra.run.smithi175.stdout:setting min_mon_release = pacific 2023-09-29T22:07:23.011 INFO:teuthology.orchestra.run.smithi175.stdout:epoch 0 2023-09-29T22:07:23.011 INFO:teuthology.orchestra.run.smithi175.stdout:fsid 0de4d642-32d1-4a19-89dd-cd5133fa56cc 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:last_changed 2023-09-29T22:07:23.011931+0000 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:created 2023-09-29T22:07:23.011931+0000 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:min_mon_release 16 (pacific) 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:election_strategy: 1 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:0: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.a 2023-09-29T22:07:23.012 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-09-29T22:07:23.013 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:07:23.067 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 0de4d642-32d1-4a19-89dd-cd5133fa56cc... 2023-09-29T22:07:23.068 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-29T22:07:23.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-29T22:07:23.157 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2023-09-29T22:07:23.157 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2023-09-29T22:07:23.157 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T22:07:23.157 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.158 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T22:07:23.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2023-09-29T22:07:23.160 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T22:07:23.161 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T22:07:23.161 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2023-09-29T22:07:23.162 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.175 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 0de4d642-32d1-4a19-89dd-cd5133fa56cc 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.163 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2023-09-29T22:07:23.164 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2023-09-29T22:07:23.165 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2023-09-29T22:07:23.166 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T22:07:23.167 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T22:07:23.168 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw sync = 20 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync data inject err probability = 0.1 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2023-09-29T22:07:23.169 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data notify interval msec = 0 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2023-09-29T22:07:23.170 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:23.171 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2023-09-29T22:07:23.171 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2023-09-29T22:07:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2023-09-29T22:07:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.175 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 0de4d642-32d1-4a19-89dd-cd5133fa56cc 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2023-09-29T22:07:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2023-09-29T22:07:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2023-09-29T22:07:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T22:07:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T22:07:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2023-09-29T22:07:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2023-09-29T22:07:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw sync = 20 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync data inject err probability = 0.1 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2023-09-29T22:07:23.183 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data notify interval msec = 0 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2023-09-29T22:07:23.184 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:23.185 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-09-29T22:07:23.186 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-09-29T22:07:23.240 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T22:07:23.241 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:23.241 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-09-29T22:07:23.255 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:23.255 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-09-29T22:07:23.310 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T22:07:23.310 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:23.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c1.keyring 2023-09-29T22:07:23.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T22:07:23.389 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:23.389 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:23.443 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T22:07:23.443 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:23.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c1.keyring 2023-09-29T22:07:23.443 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T22:07:23.483 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:23.483 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:23.537 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T22:07:23.537 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T22:07:23.538 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-09-29T22:07:23.591 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-09-29T22:07:23.594 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T22:07:23.594 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T22:07:23.594 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-09-29T22:07:23.636 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.client.0.keyring 2023-09-29T22:07:23.651 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-09-29T22:07:23.733 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.client.1.keyring 2023-09-29T22:07:23.749 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T22:07:23.749 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:07:23.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-09-29T22:07:23.815 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:23.815 INFO:tasks.ceph:role: c1.osd.0 2023-09-29T22:07:23.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T22:07:23.815 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T22:07:23.889 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:23.890 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:23.891 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:24.401 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-09-29T22:07:24.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-09-29T22:07:24.403 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-09-29T22:07:24.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-09-29T22:07:24.567 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-09-29T22:07:24.635 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:24.636 INFO:tasks.ceph:role: c1.osd.1 2023-09-29T22:07:24.636 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T22:07:24.636 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T22:07:24.710 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:24.711 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:24.712 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:24.712 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:24.712 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-09-29T22:07:25.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-09-29T22:07:25.171 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-09-29T22:07:25.265 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-09-29T22:07:25.296 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-09-29T22:07:25.363 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:25.363 INFO:tasks.ceph:role: c1.osd.2 2023-09-29T22:07:25.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T22:07:25.363 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T22:07:25.436 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:25.436 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:25.436 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:25.437 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:25.934 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2023-09-29T22:07:25.936 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2023-09-29T22:07:25.936 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-09-29T22:07:26.063 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-09-29T22:07:26.095 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.195+0000 7f97598d2500 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-09-29T22:07:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.196+0000 7f97598d2500 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-09-29T22:07:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.196+0000 7f97598d2500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T22:07:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.196+0000 7f97598d2500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T22:07:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.196+0000 7f97598d2500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T22:07:26.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:26.204+0000 7f97598d2500 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-09-29T22:07:28.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-09-29T22:07:28.320 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-09-29T22:07:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-09-29T22:07:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T22:07:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T22:07:28.411 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T22:07:28.412 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:28.411+0000 7faa60f9b500 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-09-29T22:07:30.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-09-29T22:07:30.519 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:30.609 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.608+0000 7f4a6f8a7500 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-09-29T22:07:30.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.609+0000 7f4a6f8a7500 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-09-29T22:07:30.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.609+0000 7f4a6f8a7500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T22:07:30.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.609+0000 7f4a6f8a7500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T22:07:30.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.609+0000 7f4a6f8a7500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T22:07:30.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:07:30.609+0000 7f4a6f8a7500 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-09-29T22:07:32.695 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-09-29T22:07:32.725 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T22:07:32.725 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:32.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-09-29T22:07:32.793 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:32.794 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-09-29T22:07:32.862 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:32.862 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-09-29T22:07:32.930 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:32.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-09-29T22:07:32.997 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:32.997 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-09-29T22:07:33.052 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:33.052 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-09-29T22:07:33.106 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T22:07:33.106 DEBUG:teuthology.orchestra.run.smithi175:> sudo tee -a /etc/ceph/c1.keyring 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.x] 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCbShdlnehPIxAArbicKmOn//qr+QjlgF9Rig== 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCeShdlthy5CxAAQU6AbKVdwmdI9d2u2rBzKQ== 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCgShdlUp+MGBAAaboBgn67kGEcs1ANvjkSrw== 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2023-09-29T22:07:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCiShdlNvtWJBAAsRRIHZHmscKCJjhXLfhG2g== 2023-09-29T22:07:33.173 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2023-09-29T22:07:33.173 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCbShdlLEf7JRAAOZA6HTTJU36QivXz8+DJHw== 2023-09-29T22:07:33.173 INFO:teuthology.orchestra.run.smithi175.stdout:[client.1] 2023-09-29T22:07:33.173 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQCbShdlTHrIKxAAieEekdCY+gdi+01pIzB3Rw== 2023-09-29T22:07:33.174 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T22:07:33.265 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:07:33.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:07:33.367 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:07:33.419 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T22:07:33.473 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T22:07:33.526 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T22:07:33.527 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-09-29T22:07:33.553 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-09-29T22:07:33.649 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-09-29T22:07:33.675 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-09-29T22:07:33.730 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-09-29T22:07:33.730 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-09-29T22:07:33.731 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-09-29T22:07:33.773 INFO:tasks.ceph.c1.mon.a:Started 2023-09-29T22:07:33.773 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-09-29T22:07:33.773 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-09-29T22:07:33.773 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-09-29T22:07:33.775 INFO:tasks.ceph.c1.mgr.x:Started 2023-09-29T22:07:33.776 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T22:07:33.776 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd crush tunables default 2023-09-29T22:07:33.972 INFO:teuthology.orchestra.run.smithi175.stderr:adjusted tunables profile to default 2023-09-29T22:07:33.983 INFO:tasks.ceph:check_enable_crimson: False 2023-09-29T22:07:33.983 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-09-29T22:07:33.983 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:33.983 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-09-29T22:07:34.009 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:34.009 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-09-29T22:07:34.078 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:34.078 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-09-29T22:07:34.146 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new e07ac1bf-868e-43ba-ab07-b2e973f615c3 0 2023-09-29T22:07:34.365 INFO:teuthology.orchestra.run.smithi175.stdout:0 2023-09-29T22:07:34.378 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new 4c33254a-c875-4e8f-b5ab-998beff5746f 1 2023-09-29T22:07:34.507 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:34.507+0000 7f98e7232040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T22:07:34.558 INFO:teuthology.orchestra.run.smithi175.stdout:1 2023-09-29T22:07:34.559 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:34.560+0000 7f98e7232040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T22:07:34.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new 41e609e9-f42b-4ae0-9ea0-f53e770b2ad2 2 2023-09-29T22:07:34.754 INFO:teuthology.orchestra.run.smithi175.stdout:2 2023-09-29T22:07:34.765 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-09-29T22:07:34.765 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-09-29T22:07:34.767 INFO:tasks.ceph.c1.osd.0:Started 2023-09-29T22:07:34.767 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-09-29T22:07:34.767 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-09-29T22:07:34.769 INFO:tasks.ceph.c1.osd.1:Started 2023-09-29T22:07:34.769 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-09-29T22:07:34.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-09-29T22:07:34.771 INFO:tasks.ceph.c1.osd.2:Started 2023-09-29T22:07:34.772 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:34.971 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:34.971 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:34.753625+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:34.982 INFO:tasks.ceph.ceph_manager.c1:[] 2023-09-29T22:07:34.982 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T22:07:35.148 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.148+0000 7f98e7232040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T22:07:35.198 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.199+0000 7f98e7232040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T22:07:35.242 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.243+0000 7f98e7232040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T22:07:35.284 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:35.297 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.297+0000 7f98e7232040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T22:07:35.341 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.341+0000 7f98e7232040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T22:07:35.373 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:35.373+0000 7f41a440e500 -1 Falling back to public interface 2023-09-29T22:07:35.374 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2023-09-29T22:07:35.373+0000 7f7c73218500 -1 Falling back to public interface 2023-09-29T22:07:35.374 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2023-09-29T22:07:35.374+0000 7f7c71aff640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-09-29T22:07:35.375 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:35.374+0000 7f41a2dc2640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2023-09-29T22:07:35.375 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:35.374+0000 7f41a2dc2640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6802/0: (98) Address already in use 2023-09-29T22:07:35.375 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:35.374+0000 7f41a2dc2640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6805/0: (98) Address already in use 2023-09-29T22:07:35.378 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2023-09-29T22:07:35.378+0000 7f1477614500 -1 Falling back to public interface 2023-09-29T22:07:35.444 INFO:teuthology.misc.health.smithi175.stdout: 2023-09-29T22:07:35.444 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":5,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:34.753625+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:35.449 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.449+0000 7f98e7232040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T22:07:35.454 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T22:07:35.520 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.520+0000 7f98e7232040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T22:07:35.565 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.565+0000 7f98e7232040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T22:07:35.616 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.616+0000 7f98e7232040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T22:07:35.663 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:35.663+0000 7f98e7232040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T22:07:36.038 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.038+0000 7f98e7232040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T22:07:36.170 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.170+0000 7f98e7232040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T22:07:36.298 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.299+0000 7f98e7232040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T22:07:36.344 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.344+0000 7f98e7232040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T22:07:36.441 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.441+0000 7f98e7232040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T22:07:36.536 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.536+0000 7f98e7232040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T22:07:36.673 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.673+0000 7f98e7232040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T22:07:36.715 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.714+0000 7f98e7232040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T22:07:36.854 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:36.855+0000 7f98e7232040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T22:07:37.655 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:37.654+0000 7f98e7232040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T22:07:37.928 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:37.927+0000 7f98e7232040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T22:07:37.978 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:37.977+0000 7f98e7232040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T22:07:38.078 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:38.077+0000 7f98e7232040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T22:07:38.122 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2023-09-29T22:07:38.120+0000 7f7c73218500 -1 osd.1 0 log_to_monitors true 2023-09-29T22:07:38.125 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2023-09-29T22:07:38.124+0000 7f1477614500 -1 osd.2 0 log_to_monitors true 2023-09-29T22:07:38.126 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:38.125+0000 7f41a440e500 -1 osd.0 0 log_to_monitors true 2023-09-29T22:07:39.100 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2023-09-29T22:07:39.099+0000 7f9887f10640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-29T22:07:41.304 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:41.303+0000 7f41a05bd640 -1 osd.0 0 waiting for initial osdmap 2023-09-29T22:07:41.321 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2023-09-29T22:07:41.320+0000 7f4197bc5640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:07:41.327 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2023-09-29T22:07:41.326+0000 7f14735a1640 -1 osd.2 0 waiting for initial osdmap 2023-09-29T22:07:41.335 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2023-09-29T22:07:41.335+0000 7f146b3c0640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:07:41.351 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2023-09-29T22:07:41.350+0000 7f7c6eae3640 -1 osd.1 0 waiting for initial osdmap 2023-09-29T22:07:41.366 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2023-09-29T22:07:41.365+0000 7f7c66902640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:07:41.757 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:42.024 INFO:teuthology.misc.health.smithi175.stdout: 2023-09-29T22:07:42.024 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":7,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:40.101221+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:42.035 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T22:07:48.339 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:48.614 INFO:teuthology.misc.health.smithi175.stdout: 2023-09-29T22:07:48.617 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":12,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:46.861704+0000","last_up_change":"2023-09-29T22:07:42.330131+0000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T22:07:44.137901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":82654},{"type":"v1","addr":"172.21.15.175:6803","nonce":82654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":82654},{"type":"v1","addr":"172.21.15.175:6808","nonce":82654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":82654},{"type":"v1","addr":"172.21.15.175:6815","nonce":82654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6811","nonce":82654},{"type":"v1","addr":"172.21.15.175:6813","nonce":82654}]},"public_addr":"172.21.15.175:6803/82654","cluster_addr":"172.21.15.175:6808/82654","heartbeat_back_addr":"172.21.15.175:6815/82654","heartbeat_front_addr":"172.21.15.175:6813/82654","state":["exists","up"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6801","nonce":82656},{"type":"v1","addr":"172.21.15.175:6802","nonce":82656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":82656},{"type":"v1","addr":"172.21.15.175:6805","nonce":82656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":82656},{"type":"v1","addr":"172.21.15.175:6812","nonce":82656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6807","nonce":82656},{"type":"v1","addr":"172.21.15.175:6809","nonce":82656}]},"public_addr":"172.21.15.175:6802/82656","cluster_addr":"172.21.15.175:6805/82656","heartbeat_back_addr":"172.21.15.175:6812/82656","heartbeat_front_addr":"172.21.15.175:6809/82656","state":["exists","up"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":82655},{"type":"v1","addr":"172.21.15.175:6817","nonce":82655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":82655},{"type":"v1","addr":"172.21.15.175:6819","nonce":82655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":82655},{"type":"v1","addr":"172.21.15.175:6823","nonce":82655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":82655},{"type":"v1","addr":"172.21.15.175:6821","nonce":82655}]},"public_addr":"172.21.15.175:6817/82655","cluster_addr":"172.21.15.175:6819/82655","heartbeat_back_addr":"172.21.15.175:6823/82655","heartbeat_front_addr":"172.21.15.175:6821/82655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.161452+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.111386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.102479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:48.626 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-29T22:07:48.626 INFO:tasks.ceph:Creating RBD pool 2023-09-29T22:07:48.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd pool create rbd 8 2023-09-29T22:07:49.869 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'rbd' created 2023-09-29T22:07:49.886 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T22:07:50.870 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T22:07:50.887 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-09-29T22:07:50.888 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-09-29T22:07:50.888 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-09-29T22:07:50.888 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-09-29T22:07:50.888 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T22:07:51.217 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:51.228 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":82318},{"type":"v1","addr":"172.21.15.175:6825","nonce":82318}]},"active_addr":"172.21.15.175:6825/82318","active_change":"2023-09-29T22:07:38.091488+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":2556485948}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":1020094066}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":1767387369}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":2528352259}]}]} 2023-09-29T22:07:51.231 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-09-29T22:07:51.231 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-09-29T22:07:51.231 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:51.509 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:51.509 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":14,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:50.867646+0000","last_up_change":"2023-09-29T22:07:42.330131+0000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T22:07:44.137901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T22:07:48.919789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":82654},{"type":"v1","addr":"172.21.15.175:6803","nonce":82654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":82654},{"type":"v1","addr":"172.21.15.175:6808","nonce":82654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":82654},{"type":"v1","addr":"172.21.15.175:6815","nonce":82654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6811","nonce":82654},{"type":"v1","addr":"172.21.15.175:6813","nonce":82654}]},"public_addr":"172.21.15.175:6803/82654","cluster_addr":"172.21.15.175:6808/82654","heartbeat_back_addr":"172.21.15.175:6815/82654","heartbeat_front_addr":"172.21.15.175:6813/82654","state":["exists","up"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6801","nonce":82656},{"type":"v1","addr":"172.21.15.175:6802","nonce":82656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":82656},{"type":"v1","addr":"172.21.15.175:6805","nonce":82656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":82656},{"type":"v1","addr":"172.21.15.175:6812","nonce":82656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6807","nonce":82656},{"type":"v1","addr":"172.21.15.175:6809","nonce":82656}]},"public_addr":"172.21.15.175:6802/82656","cluster_addr":"172.21.15.175:6805/82656","heartbeat_back_addr":"172.21.15.175:6812/82656","heartbeat_front_addr":"172.21.15.175:6809/82656","state":["exists","up"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":82655},{"type":"v1","addr":"172.21.15.175:6817","nonce":82655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":82655},{"type":"v1","addr":"172.21.15.175:6819","nonce":82655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":82655},{"type":"v1","addr":"172.21.15.175:6823","nonce":82655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":82655},{"type":"v1","addr":"172.21.15.175:6821","nonce":82655}]},"public_addr":"172.21.15.175:6817/82655","cluster_addr":"172.21.15.175:6819/82655","heartbeat_back_addr":"172.21.15.175:6823/82655","heartbeat_front_addr":"172.21.15.175:6821/82655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.161452+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.111386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.102479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:51.521 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-09-29T22:07:51.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T22:07:51.793 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:51.794 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":14,"fsid":"0de4d642-32d1-4a19-89dd-cd5133fa56cc","created":"2023-09-29T22:07:33.894855+0000","modified":"2023-09-29T22:07:50.867646+0000","last_up_change":"2023-09-29T22:07:42.330131+0000","last_in_change":"2023-09-29T22:07:34.753625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T22:07:44.137901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T22:07:48.919789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e07ac1bf-868e-43ba-ab07-b2e973f615c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":82654},{"type":"v1","addr":"172.21.15.175:6803","nonce":82654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":82654},{"type":"v1","addr":"172.21.15.175:6808","nonce":82654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":82654},{"type":"v1","addr":"172.21.15.175:6815","nonce":82654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6811","nonce":82654},{"type":"v1","addr":"172.21.15.175:6813","nonce":82654}]},"public_addr":"172.21.15.175:6803/82654","cluster_addr":"172.21.15.175:6808/82654","heartbeat_back_addr":"172.21.15.175:6815/82654","heartbeat_front_addr":"172.21.15.175:6813/82654","state":["exists","up"]},{"osd":1,"uuid":"4c33254a-c875-4e8f-b5ab-998beff5746f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6801","nonce":82656},{"type":"v1","addr":"172.21.15.175:6802","nonce":82656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":82656},{"type":"v1","addr":"172.21.15.175:6805","nonce":82656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":82656},{"type":"v1","addr":"172.21.15.175:6812","nonce":82656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6807","nonce":82656},{"type":"v1","addr":"172.21.15.175:6809","nonce":82656}]},"public_addr":"172.21.15.175:6802/82656","cluster_addr":"172.21.15.175:6805/82656","heartbeat_back_addr":"172.21.15.175:6812/82656","heartbeat_front_addr":"172.21.15.175:6809/82656","state":["exists","up"]},{"osd":2,"uuid":"41e609e9-f42b-4ae0-9ea0-f53e770b2ad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":82655},{"type":"v1","addr":"172.21.15.175:6817","nonce":82655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":82655},{"type":"v1","addr":"172.21.15.175:6819","nonce":82655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":82655},{"type":"v1","addr":"172.21.15.175:6823","nonce":82655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":82655},{"type":"v1","addr":"172.21.15.175:6821","nonce":82655}]},"public_addr":"172.21.15.175:6817/82655","cluster_addr":"172.21.15.175:6819/82655","heartbeat_back_addr":"172.21.15.175:6823/82655","heartbeat_front_addr":"172.21.15.175:6821/82655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.161452+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.111386+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:07:39.102479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:07:51.806 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-09-29T22:07:51.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-09-29T22:07:51.807 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-09-29T22:07:51.943 INFO:teuthology.orchestra.run.smithi175.stdout:34359738374 2023-09-29T22:07:51.944 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-29T22:07:51.965 INFO:teuthology.orchestra.run.smithi175.stdout:34359738373 2023-09-29T22:07:51.965 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-29T22:07:51.966 INFO:teuthology.orchestra.run.smithi175.stdout:34359738372 2023-09-29T22:07:51.967 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-29T22:07:52.253 INFO:teuthology.orchestra.run.smithi175.stdout:34359738373 2023-09-29T22:07:52.266 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738373 for osd.0 2023-09-29T22:07:52.305 INFO:teuthology.orchestra.run.smithi175.stdout:34359738372 2023-09-29T22:07:52.320 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738372 for osd.1 2023-09-29T22:07:52.324 INFO:teuthology.orchestra.run.smithi175.stdout:34359738371 2023-09-29T22:07:52.337 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.2 2023-09-29T22:07:53.267 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-29T22:07:53.321 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-29T22:07:53.338 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-29T22:07:53.575 INFO:teuthology.orchestra.run.smithi175.stdout:34359738374 2023-09-29T22:07:53.588 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.0 2023-09-29T22:07:53.588 DEBUG:teuthology.parallel:result is None 2023-09-29T22:07:53.621 INFO:teuthology.orchestra.run.smithi175.stdout:34359738373 2023-09-29T22:07:53.633 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.1 2023-09-29T22:07:53.634 DEBUG:teuthology.parallel:result is None 2023-09-29T22:07:53.657 INFO:teuthology.orchestra.run.smithi175.stdout:34359738372 2023-09-29T22:07:53.669 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2023-09-29T22:07:53.670 DEBUG:teuthology.parallel:result is None 2023-09-29T22:07:53.670 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-09-29T22:07:53.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T22:07:53.986 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:53.987 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-09-29T22:07:53.998 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-09-29T22:07:52.097307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82376,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490424,"statfs":{"total":322122547200,"available":322038194176,"internally_reserved":0,"allocated":1556480,"data_stored":1281896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.228720"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.883879+0000","last_change":"2023-09-29T22:07:50.879939+0000","last_active":"2023-09-29T22:07:50.883879+0000","last_peered":"2023-09-29T22:07:50.883879+0000","last_clean":"2023-09-29T22:07:50.883879+0000","last_became_active":"2023-09-29T22:07:50.879399+0000","last_became_peered":"2023-09-29T22:07:50.879399+0000","last_unstale":"2023-09-29T22:07:50.883879+0000","last_undegraded":"2023-09-29T22:07:50.883879+0000","last_fullsized":"2023-09-29T22:07:50.883879+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:05:01.540367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.883867+0000","last_change":"2023-09-29T22:07:50.880176+0000","last_active":"2023-09-29T22:07:50.883867+0000","last_peered":"2023-09-29T22:07:50.883867+0000","last_clean":"2023-09-29T22:07:50.883867+0000","last_became_active":"2023-09-29T22:07:50.879925+0000","last_became_peered":"2023-09-29T22:07:50.879925+0000","last_unstale":"2023-09-29T22:07:50.883867+0000","last_undegraded":"2023-09-29T22:07:50.883867+0000","last_fullsized":"2023-09-29T22:07:50.883867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:50.850879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.883889+0000","last_change":"2023-09-29T22:07:50.880347+0000","last_active":"2023-09-29T22:07:50.883889+0000","last_peered":"2023-09-29T22:07:50.883889+0000","last_clean":"2023-09-29T22:07:50.883889+0000","last_became_active":"2023-09-29T22:07:50.879858+0000","last_became_peered":"2023-09-29T22:07:50.879858+0000","last_unstale":"2023-09-29T22:07:50.883889+0000","last_undegraded":"2023-09-29T22:07:50.883889+0000","last_fullsized":"2023-09-29T22:07:50.883889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:21:42.405040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.883884+0000","last_change":"2023-09-29T22:07:50.880309+0000","last_active":"2023-09-29T22:07:50.883884+0000","last_peered":"2023-09-29T22:07:50.883884+0000","last_clean":"2023-09-29T22:07:50.883884+0000","last_became_active":"2023-09-29T22:07:50.879448+0000","last_became_peered":"2023-09-29T22:07:50.879448+0000","last_unstale":"2023-09-29T22:07:50.883884+0000","last_undegraded":"2023-09-29T22:07:50.883884+0000","last_fullsized":"2023-09-29T22:07:50.883884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:21:57.421592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.883556+0000","last_change":"2023-09-29T22:07:50.878969+0000","last_active":"2023-09-29T22:07:51.883556+0000","last_peered":"2023-09-29T22:07:51.883556+0000","last_clean":"2023-09-29T22:07:51.883556+0000","last_became_active":"2023-09-29T22:07:50.878715+0000","last_became_peered":"2023-09-29T22:07:50.878715+0000","last_unstale":"2023-09-29T22:07:51.883556+0000","last_undegraded":"2023-09-29T22:07:51.883556+0000","last_fullsized":"2023-09-29T22:07:51.883556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:13:22.457490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.881076+0000","last_change":"2023-09-29T22:07:50.879084+0000","last_active":"2023-09-29T22:07:50.881076+0000","last_peered":"2023-09-29T22:07:50.881076+0000","last_clean":"2023-09-29T22:07:50.881076+0000","last_became_active":"2023-09-29T22:07:50.878839+0000","last_became_peered":"2023-09-29T22:07:50.878839+0000","last_unstale":"2023-09-29T22:07:50.881076+0000","last_undegraded":"2023-09-29T22:07:50.881076+0000","last_fullsized":"2023-09-29T22:07:50.881076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:13:22.457490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.881059+0000","last_change":"2023-09-29T22:07:50.879052+0000","last_active":"2023-09-29T22:07:50.881059+0000","last_peered":"2023-09-29T22:07:50.881059+0000","last_clean":"2023-09-29T22:07:50.881059+0000","last_became_active":"2023-09-29T22:07:50.878785+0000","last_became_peered":"2023-09-29T22:07:50.878785+0000","last_unstale":"2023-09-29T22:07:50.881059+0000","last_undegraded":"2023-09-29T22:07:50.881059+0000","last_fullsized":"2023-09-29T22:07:50.881059+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:03:32.849359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.880951+0000","last_change":"2023-09-29T22:07:50.878908+0000","last_active":"2023-09-29T22:07:50.880951+0000","last_peered":"2023-09-29T22:07:50.880951+0000","last_clean":"2023-09-29T22:07:50.880951+0000","last_became_active":"2023-09-29T22:07:50.878624+0000","last_became_peered":"2023-09-29T22:07:50.878624+0000","last_unstale":"2023-09-29T22:07:50.880951+0000","last_undegraded":"2023-09-29T22:07:50.880951+0000","last_fullsized":"2023-09-29T22:07:50.880951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:10:39.834224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-29T22:07:49.869619+0000","last_change":"2023-09-29T22:07:46.153139+0000","last_active":"2023-09-29T22:07:49.869619+0000","last_peered":"2023-09-29T22:07:49.869619+0000","last_clean":"2023-09-29T22:07:49.869619+0000","last_became_active":"2023-09-29T22:07:46.152218+0000","last_became_peered":"2023-09-29T22:07:46.152218+0000","last_unstale":"2023-09-29T22:07:49.869619+0000","last_undegraded":"2023-09-29T22:07:49.869619+0000","last_fullsized":"2023-09-29T22:07:49.869619+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:44.856481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:44.856481+0000","last_clean_scrub_stamp":"2023-09-29T22:07:44.856481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:38:39.998518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T22:07:53.999 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T22:07:54.272 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:54.272 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2023-09-29T22:07:54.283 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-09-29T22:07:54.097594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82392,"kb_used_data":1536,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490408,"statfs":{"total":322122547200,"available":322038177792,"internally_reserved":0,"allocated":1572864,"data_stored":1288852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.229007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.950045+0000","last_change":"2023-09-29T22:07:50.879939+0000","last_active":"2023-09-29T22:07:51.950045+0000","last_peered":"2023-09-29T22:07:51.950045+0000","last_clean":"2023-09-29T22:07:51.950045+0000","last_became_active":"2023-09-29T22:07:50.879399+0000","last_became_peered":"2023-09-29T22:07:50.879399+0000","last_unstale":"2023-09-29T22:07:51.950045+0000","last_undegraded":"2023-09-29T22:07:51.950045+0000","last_fullsized":"2023-09-29T22:07:51.950045+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:05:01.540367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.950034+0000","last_change":"2023-09-29T22:07:50.880176+0000","last_active":"2023-09-29T22:07:51.950034+0000","last_peered":"2023-09-29T22:07:51.950034+0000","last_clean":"2023-09-29T22:07:51.950034+0000","last_became_active":"2023-09-29T22:07:50.879925+0000","last_became_peered":"2023-09-29T22:07:50.879925+0000","last_unstale":"2023-09-29T22:07:51.950034+0000","last_undegraded":"2023-09-29T22:07:51.950034+0000","last_fullsized":"2023-09-29T22:07:51.950034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:50.850879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.950025+0000","last_change":"2023-09-29T22:07:50.880347+0000","last_active":"2023-09-29T22:07:51.950025+0000","last_peered":"2023-09-29T22:07:51.950025+0000","last_clean":"2023-09-29T22:07:51.950025+0000","last_became_active":"2023-09-29T22:07:50.879858+0000","last_became_peered":"2023-09-29T22:07:50.879858+0000","last_unstale":"2023-09-29T22:07:51.950025+0000","last_undegraded":"2023-09-29T22:07:51.950025+0000","last_fullsized":"2023-09-29T22:07:51.950025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:21:42.405040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.950016+0000","last_change":"2023-09-29T22:07:50.880309+0000","last_active":"2023-09-29T22:07:51.950016+0000","last_peered":"2023-09-29T22:07:51.950016+0000","last_clean":"2023-09-29T22:07:51.950016+0000","last_became_active":"2023-09-29T22:07:50.879448+0000","last_became_peered":"2023-09-29T22:07:50.879448+0000","last_unstale":"2023-09-29T22:07:51.950016+0000","last_undegraded":"2023-09-29T22:07:51.950016+0000","last_fullsized":"2023-09-29T22:07:51.950016+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:21:57.421592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.883556+0000","last_change":"2023-09-29T22:07:50.878969+0000","last_active":"2023-09-29T22:07:51.883556+0000","last_peered":"2023-09-29T22:07:51.883556+0000","last_clean":"2023-09-29T22:07:51.883556+0000","last_became_active":"2023-09-29T22:07:50.878715+0000","last_became_peered":"2023-09-29T22:07:50.878715+0000","last_unstale":"2023-09-29T22:07:51.883556+0000","last_undegraded":"2023-09-29T22:07:51.883556+0000","last_fullsized":"2023-09-29T22:07:51.883556+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:13:22.457490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.881076+0000","last_change":"2023-09-29T22:07:50.879084+0000","last_active":"2023-09-29T22:07:50.881076+0000","last_peered":"2023-09-29T22:07:50.881076+0000","last_clean":"2023-09-29T22:07:50.881076+0000","last_became_active":"2023-09-29T22:07:50.878839+0000","last_became_peered":"2023-09-29T22:07:50.878839+0000","last_unstale":"2023-09-29T22:07:50.881076+0000","last_undegraded":"2023-09-29T22:07:50.881076+0000","last_fullsized":"2023-09-29T22:07:50.881076+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:13:22.457490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.881059+0000","last_change":"2023-09-29T22:07:50.879052+0000","last_active":"2023-09-29T22:07:50.881059+0000","last_peered":"2023-09-29T22:07:50.881059+0000","last_clean":"2023-09-29T22:07:50.881059+0000","last_became_active":"2023-09-29T22:07:50.878785+0000","last_became_peered":"2023-09-29T22:07:50.878785+0000","last_unstale":"2023-09-29T22:07:50.881059+0000","last_undegraded":"2023-09-29T22:07:50.881059+0000","last_fullsized":"2023-09-29T22:07:50.881059+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:03:32.849359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:51.950001+0000","last_change":"2023-09-29T22:07:50.878908+0000","last_active":"2023-09-29T22:07:51.950001+0000","last_peered":"2023-09-29T22:07:51.950001+0000","last_clean":"2023-09-29T22:07:51.950001+0000","last_became_active":"2023-09-29T22:07:50.878624+0000","last_became_peered":"2023-09-29T22:07:50.878624+0000","last_unstale":"2023-09-29T22:07:51.950001+0000","last_undegraded":"2023-09-29T22:07:51.950001+0000","last_fullsized":"2023-09-29T22:07:51.950001+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:49.866635+0000","last_clean_scrub_stamp":"2023-09-29T22:07:49.866635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:10:39.834224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-29T22:07:50.870373+0000","last_change":"2023-09-29T22:07:46.153139+0000","last_active":"2023-09-29T22:07:50.870373+0000","last_peered":"2023-09-29T22:07:50.870373+0000","last_clean":"2023-09-29T22:07:50.870373+0000","last_became_active":"2023-09-29T22:07:46.152218+0000","last_became_peered":"2023-09-29T22:07:46.152218+0000","last_unstale":"2023-09-29T22:07:50.870373+0000","last_undegraded":"2023-09-29T22:07:50.870373+0000","last_fullsized":"2023-09-29T22:07:50.870373+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:07:44.856481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:07:44.856481+0000","last_clean_scrub_stamp":"2023-09-29T22:07:44.856481+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:38:39.998518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27660,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27660,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T22:07:54.284 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-09-29T22:07:54.285 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-09-29T22:07:54.285 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-09-29T22:07:54.285 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-09-29T22:07:54.591 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:54.591 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-29T22:07:54.603 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-09-29T22:07:54.603 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T22:07:54.612 INFO:tasks.ceph:Creating ceph cluster c2... 2023-09-29T22:07:54.612 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T22:07:54.612 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406303', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4212, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'targets': {'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGg1Rj0/eERk8ONdHWi2h+CdqkStQ5MR5QKLDJW4g0aqKq3EScLjXaZ/6ZMMiGzDXOObRTOT0dqrZkf9wcsjTQk=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjK/BJMklETggsMW2i4eJ/4RHkZRjVcDsj06GMnEus1nm4X8DpnPcTSVd41NffPZrjXOA+ewpgniZENFjXO//c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_12:36:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T22:07:54.613 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-29T22:07:54.616 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-29T22:07:54.634 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T22:07:54.671 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T22:07:54.705 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:54.705 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T22:07:54.763 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T22:07:54.763 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2023-09-29T22:07:54.819 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-09-29 22:07:17.182441390 +0000 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-09-29 22:04:49.170056090 +0000 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-09-29 22:04:49.170056090 +0000 2023-09-29T22:07:54.820 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-09-29 22:04:49.170056090 +0000 2023-09-29T22:07:54.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T22:07:54.888 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-09-29T22:07:54.888 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-09-29T22:07:54.888 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000212525 s, 2.4 MB/s 2023-09-29T22:07:54.889 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T22:07:54.946 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-09-29 22:07:17.183441370 +0000 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-09-29 22:04:49.689045603 +0000 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-09-29 22:04:49.689045603 +0000 2023-09-29T22:07:55.002 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-09-29 22:04:49.689045603 +0000 2023-09-29T22:07:55.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T22:07:55.068 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-09-29T22:07:55.068 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-09-29T22:07:55.068 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000193751 s, 2.6 MB/s 2023-09-29T22:07:55.069 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T22:07:55.124 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-09-29 22:07:17.183441370 +0000 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-09-29 22:04:50.176035764 +0000 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-09-29 22:04:50.176035764 +0000 2023-09-29T22:07:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-09-29 22:04:50.176035764 +0000 2023-09-29T22:07:55.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T22:07:55.246 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-09-29T22:07:55.246 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-09-29T22:07:55.246 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000204974 s, 2.5 MB/s 2023-09-29T22:07:55.247 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T22:07:55.302 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2023-09-29 22:07:17.183441370 +0000 2023-09-29T22:07:55.358 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2023-09-29 22:04:50.667025844 +0000 2023-09-29T22:07:55.359 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2023-09-29 22:04:50.667025844 +0000 2023-09-29T22:07:55.359 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2023-09-29 22:04:50.667025844 +0000 2023-09-29T22:07:55.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T22:07:55.424 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2023-09-29T22:07:55.424 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2023-09-29T22:07:55.424 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000232684 s, 2.2 MB/s 2023-09-29T22:07:55.425 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T22:07:55.481 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:55.481 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:07:55.481 INFO:tasks.ceph:Generating config... 2023-09-29T22:07:55.482 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-29T22:07:55.482 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-29T22:07:55.483 INFO:tasks.ceph:[client] setuser = ceph 2023-09-29T22:07:55.484 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-29T22:07:55.484 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T22:07:55.484 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T22:07:55.484 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T22:07:55.484 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T22:07:55.484 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:55.484 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:55.485 INFO:tasks.ceph:Setting up c2.mon.a... 2023-09-29T22:07:55.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-09-29T22:07:55.567 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.keyring 2023-09-29T22:07:55.570 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-09-29T22:07:55.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T22:07:55.679 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.184')] 2023-09-29T22:07:55.680 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.184'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-29T22:07:55.680 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:55.681 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:07:55.733 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.184 --print /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:07:55.810 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:55.810 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: generated fsid 713bf48e-c521-49ea-a8a1-6dd1a1ff10c6 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:setting min_mon_release = pacific 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:epoch 0 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:fsid 713bf48e-c521-49ea-a8a1-6dd1a1ff10c6 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:last_changed 2023-09-29T22:07:55.811510+0000 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:created 2023-09-29T22:07:55.811510+0000 2023-09-29T22:07:55.811 INFO:teuthology.orchestra.run.smithi184.stdout:min_mon_release 16 (pacific) 2023-09-29T22:07:55.812 INFO:teuthology.orchestra.run.smithi184.stdout:election_strategy: 1 2023-09-29T22:07:55.812 INFO:teuthology.orchestra.run.smithi184.stdout:0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2023-09-29T22:07:55.812 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-09-29T22:07:55.813 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T22:07:55.868 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 713bf48e-c521-49ea-a8a1-6dd1a1ff10c6... 2023-09-29T22:07:55.868 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-29T22:07:55.871 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-29T22:07:55.926 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2023-09-29T22:07:55.927 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T22:07:55.928 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T22:07:55.929 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2023-09-29T22:07:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.184 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 713bf48e-c521-49ea-a8a1-6dd1a1ff10c6 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2023-09-29T22:07:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2023-09-29T22:07:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2023-09-29T22:07:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw sync = 20 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync data inject err probability = 0.1 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data notify interval msec = 0 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2023-09-29T22:07:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T22:07:55.939 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2023-09-29T22:07:55.939 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:55.939 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2023-09-29T22:07:55.939 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:55.958 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2023-09-29T22:07:55.958 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2023-09-29T22:07:55.958 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T22:07:55.958 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T22:07:55.959 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2023-09-29T22:07:55.960 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.961 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.184 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 713bf48e-c521-49ea-a8a1-6dd1a1ff10c6 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2023-09-29T22:07:55.962 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.963 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2023-09-29T22:07:55.964 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2023-09-29T22:07:55.965 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T22:07:55.966 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2023-09-29T22:07:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw sync = 20 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync data inject err probability = 0.1 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2023-09-29T22:07:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data notify interval msec = 0 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2023-09-29T22:07:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T22:07:55.972 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-09-29T22:07:55.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-09-29T22:07:56.067 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T22:07:56.067 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:56.067 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-09-29T22:07:56.082 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:56.083 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-09-29T22:07:56.137 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2023-09-29T22:07:56.137 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:56.138 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c2.keyring 2023-09-29T22:07:56.138 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T22:07:56.181 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2023-09-29T22:07:56.182 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-29T22:07:56.237 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T22:07:56.237 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:56.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c2.keyring 2023-09-29T22:07:56.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T22:07:56.277 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:07:56.277 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-29T22:07:56.332 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T22:07:56.332 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T22:07:56.332 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-09-29T22:07:56.426 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-09-29T22:07:56.428 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T22:07:56.428 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T22:07:56.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-09-29T22:07:56.470 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.0.keyring 2023-09-29T22:07:56.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-09-29T22:07:56.566 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.1.keyring 2023-09-29T22:07:56.582 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T22:07:56.582 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T22:07:56.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-09-29T22:07:56.649 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:56.649 INFO:tasks.ceph:role: c2.osd.0 2023-09-29T22:07:56.649 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T22:07:56.649 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:56.723 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:57.244 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-09-29T22:07:57.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-09-29T22:07:57.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-09-29T22:07:57.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-09-29T22:07:57.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-09-29T22:07:57.468 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:57.468 INFO:tasks.ceph:role: c2.osd.1 2023-09-29T22:07:57.468 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T22:07:57.468 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T22:07:57.542 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:57.542 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:57.543 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:57.543 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:57.543 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:57.994 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-09-29T22:07:57.996 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-09-29T22:07:57.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-09-29T22:07:58.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-09-29T22:07:58.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-09-29T22:07:58.189 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T22:07:58.189 INFO:tasks.ceph:role: c2.osd.2 2023-09-29T22:07:58.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-29T22:07:58.189 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T22:07:58.263 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T22:07:58.263 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T22:07:58.264 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T22:07:58.750 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2023-09-29T22:07:58.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2023-09-29T22:07:58.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-09-29T22:07:58.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-09-29T22:07:58.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:07:59.033 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.033+0000 7f2382a0e500 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-09-29T22:07:59.039 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.039+0000 7f2382a0e500 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-09-29T22:07:59.039 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.039+0000 7f2382a0e500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T22:07:59.039 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.039+0000 7f2382a0e500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T22:07:59.039 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.039+0000 7f2382a0e500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T22:07:59.040 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:07:59.040+0000 7f2382a0e500 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-09-29T22:08:01.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-09-29T22:08:01.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:08:01.253 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.252+0000 7fd4c461d500 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-09-29T22:08:01.253 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.253+0000 7fd4c461d500 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-09-29T22:08:01.253 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.253+0000 7fd4c461d500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T22:08:01.254 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.253+0000 7fd4c461d500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T22:08:01.254 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.253+0000 7fd4c461d500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T22:08:01.254 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:01.253+0000 7fd4c461d500 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-09-29T22:08:03.328 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-09-29T22:08:03.359 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:08:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.451+0000 7f5345d9b500 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-09-29T22:08:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.451+0000 7f5345d9b500 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-09-29T22:08:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.451+0000 7f5345d9b500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T22:08:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.451+0000 7f5345d9b500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T22:08:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.451+0000 7f5345d9b500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T22:08:03.453 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:03.452+0000 7f5345d9b500 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-09-29T22:08:05.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-09-29T22:08:05.572 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T22:08:05.572 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.572 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-09-29T22:08:05.644 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.644 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-09-29T22:08:05.712 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-09-29T22:08:05.781 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-09-29T22:08:05.849 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.849 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-09-29T22:08:05.904 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:05.904 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-09-29T22:08:05.959 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T22:08:05.959 DEBUG:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/c2.keyring 2023-09-29T22:08:06.024 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC8ShdlfDh2GRAA0VdV37+cBtLgzSN3M03swA== 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC/ShdlCo8RAhAAlcTCnN2+Wk6oKHeISnWRNQ== 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDBShdlrVshDxAAQtzKF96OZJFTdZIKXvPPzw== 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDDShdl6nD7GhAA1CBAkU5NHNzoyMpWQpM+jA== 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC8ShdlFoAXHBAA9oqbEsC+O5lRbToLR5HBKQ== 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2023-09-29T22:08:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC8Shdl+gTYIRAAwAWuCsrm37UXOuIDmTObpw== 2023-09-29T22:08:06.026 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T22:08:06.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:08:06.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:08:06.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T22:08:06.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T22:08:06.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T22:08:06.383 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T22:08:06.383 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-09-29T22:08:06.409 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-09-29T22:08:06.508 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-09-29T22:08:06.534 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-09-29T22:08:06.589 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-09-29T22:08:06.590 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-09-29T22:08:06.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-09-29T22:08:06.631 INFO:tasks.ceph.c2.mon.a:Started 2023-09-29T22:08:06.631 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-09-29T22:08:06.632 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-09-29T22:08:06.632 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-09-29T22:08:06.634 INFO:tasks.ceph.c2.mgr.x:Started 2023-09-29T22:08:06.634 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T22:08:06.634 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd crush tunables default 2023-09-29T22:08:06.828 INFO:teuthology.orchestra.run.smithi184.stderr:adjusted tunables profile to default 2023-09-29T22:08:06.839 INFO:tasks.ceph:check_enable_crimson: False 2023-09-29T22:08:06.839 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-09-29T22:08:06.839 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:06.839 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-09-29T22:08:06.866 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:06.866 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-09-29T22:08:06.934 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2023-09-29T22:08:06.934 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-09-29T22:08:07.001 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new a5916e3e-d36e-4088-b2e2-226a39dcc4a5 0 2023-09-29T22:08:07.222 INFO:teuthology.orchestra.run.smithi184.stdout:0 2023-09-29T22:08:07.233 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new b2ec478a-203e-4598-a7d6-6e6687417b6c 1 2023-09-29T22:08:07.414 INFO:teuthology.orchestra.run.smithi184.stdout:1 2023-09-29T22:08:07.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new d4d6c3a7-c7c3-428a-8695-7ba22e5af108 2 2023-09-29T22:08:07.426 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:07.426+0000 7f6824c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T22:08:07.481 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:07.480+0000 7f6824c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T22:08:07.607 INFO:teuthology.orchestra.run.smithi184.stdout:2 2023-09-29T22:08:07.618 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-09-29T22:08:07.618 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-09-29T22:08:07.620 INFO:tasks.ceph.c2.osd.0:Started 2023-09-29T22:08:07.620 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-09-29T22:08:07.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-09-29T22:08:07.622 INFO:tasks.ceph.c2.osd.1:Started 2023-09-29T22:08:07.622 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-09-29T22:08:07.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-09-29T22:08:07.624 INFO:tasks.ceph.c2.osd.2:Started 2023-09-29T22:08:07.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T22:08:07.797 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:07.797 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":5,"fsid":"713bf48e-c521-49ea-a8a1-6dd1a1ff10c6","created":"2023-09-29T22:08:06.750233+0000","modified":"2023-09-29T22:08:07.606219+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T22:08:07.606219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5916e3e-d36e-4088-b2e2-226a39dcc4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2ec478a-203e-4598-a7d6-6e6687417b6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4d6c3a7-c7c3-428a-8695-7ba22e5af108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:08:07.808 INFO:tasks.ceph.ceph_manager.c2:[] 2023-09-29T22:08:07.808 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T22:08:08.059 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.059+0000 7f6824c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T22:08:08.110 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-29T22:08:08.110 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.110+0000 7f6824c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T22:08:08.157 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.157+0000 7f6824c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T22:08:08.213 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.213+0000 7f6824c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T22:08:08.219 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2023-09-29T22:08:08.219+0000 7f5cdc42a500 -1 Falling back to public interface 2023-09-29T22:08:08.222 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2023-09-29T22:08:08.222+0000 7fd801ad4500 -1 Falling back to public interface 2023-09-29T22:08:08.240 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2023-09-29T22:08:08.240+0000 7f8ce569a500 -1 Falling back to public interface 2023-09-29T22:08:08.258 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.258+0000 7f6824c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T22:08:08.270 INFO:teuthology.misc.health.smithi184.stdout: 2023-09-29T22:08:08.270 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":5,"fsid":"713bf48e-c521-49ea-a8a1-6dd1a1ff10c6","created":"2023-09-29T22:08:06.750233+0000","modified":"2023-09-29T22:08:07.606219+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T22:08:07.606219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5916e3e-d36e-4088-b2e2-226a39dcc4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2ec478a-203e-4598-a7d6-6e6687417b6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4d6c3a7-c7c3-428a-8695-7ba22e5af108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:08:08.280 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T22:08:08.361 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.360+0000 7f6824c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T22:08:08.429 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.428+0000 7f6824c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T22:08:08.474 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.474+0000 7f6824c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T22:08:08.528 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.527+0000 7f6824c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T22:08:08.573 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.573+0000 7f6824c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T22:08:08.948 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:08.948+0000 7f6824c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T22:08:09.082 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.082+0000 7f6824c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T22:08:09.196 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.196+0000 7f6824c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T22:08:09.240 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.240+0000 7f6824c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T22:08:09.337 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.337+0000 7f6824c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T22:08:09.430 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.429+0000 7f6824c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T22:08:09.566 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.565+0000 7f6824c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T22:08:09.609 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.609+0000 7f6824c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T22:08:09.746 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:09.746+0000 7f6824c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T22:08:10.542 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:10.542+0000 7f6824c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T22:08:10.814 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:10.813+0000 7f6824c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T22:08:10.864 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:10.863+0000 7f6824c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T22:08:10.968 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2023-09-29T22:08:10.968+0000 7f5cdc42a500 -1 osd.0 0 log_to_monitors true 2023-09-29T22:08:10.969 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:10.969+0000 7f6824c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T22:08:10.978 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2023-09-29T22:08:10.978+0000 7fd801ad4500 -1 osd.2 0 log_to_monitors true 2023-09-29T22:08:10.995 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2023-09-29T22:08:10.995+0000 7f8ce569a500 -1 osd.1 0 log_to_monitors true 2023-09-29T22:08:11.990 INFO:tasks.ceph.c2.mgr.x.smithi184.stderr:2023-09-29T22:08:11.989+0000 7f67c61f4640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-29T22:08:13.210 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2023-09-29T22:08:13.209+0000 7fd7fda5f640 -1 osd.2 0 waiting for initial osdmap 2023-09-29T22:08:13.224 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2023-09-29T22:08:13.223+0000 7fd7f587e640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:08:13.259 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2023-09-29T22:08:13.259+0000 7f5cd795e640 -1 osd.0 0 waiting for initial osdmap 2023-09-29T22:08:13.275 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2023-09-29T22:08:13.275+0000 7f5ccf77d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:08:13.300 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2023-09-29T22:08:13.300+0000 7f8ce1e3c640 -1 osd.1 0 waiting for initial osdmap 2023-09-29T22:08:13.310 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2023-09-29T22:08:13.310+0000 7f8cd9444640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T22:08:14.584 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-29T22:08:14.861 INFO:teuthology.misc.health.smithi184.stdout: 2023-09-29T22:08:14.861 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":8,"fsid":"713bf48e-c521-49ea-a8a1-6dd1a1ff10c6","created":"2023-09-29T22:08:06.750233+0000","modified":"2023-09-29T22:08:14.212208+0000","last_up_change":"2023-09-29T22:08:14.212208+0000","last_in_change":"2023-09-29T22:08:07.606219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5916e3e-d36e-4088-b2e2-226a39dcc4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":82781},{"type":"v1","addr":"172.21.15.184:6801","nonce":82781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":82781},{"type":"v1","addr":"172.21.15.184:6803","nonce":82781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":82781},{"type":"v1","addr":"172.21.15.184:6807","nonce":82781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":82781},{"type":"v1","addr":"172.21.15.184:6805","nonce":82781}]},"public_addr":"172.21.15.184:6801/82781","cluster_addr":"172.21.15.184:6803/82781","heartbeat_back_addr":"172.21.15.184:6807/82781","heartbeat_front_addr":"172.21.15.184:6805/82781","state":["exists","up"]},{"osd":1,"uuid":"b2ec478a-203e-4598-a7d6-6e6687417b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":82783},{"type":"v1","addr":"172.21.15.184:6817","nonce":82783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":82783},{"type":"v1","addr":"172.21.15.184:6819","nonce":82783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":82783},{"type":"v1","addr":"172.21.15.184:6823","nonce":82783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":82783},{"type":"v1","addr":"172.21.15.184:6821","nonce":82783}]},"public_addr":"172.21.15.184:6817/82783","cluster_addr":"172.21.15.184:6819/82783","heartbeat_back_addr":"172.21.15.184:6823/82783","heartbeat_front_addr":"172.21.15.184:6821/82783","state":["exists","up"]},{"osd":2,"uuid":"d4d6c3a7-c7c3-428a-8695-7ba22e5af108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":82782},{"type":"v1","addr":"172.21.15.184:6809","nonce":82782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":82782},{"type":"v1","addr":"172.21.15.184:6811","nonce":82782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":82782},{"type":"v1","addr":"172.21.15.184:6815","nonce":82782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":82782},{"type":"v1","addr":"172.21.15.184:6813","nonce":82782}]},"public_addr":"172.21.15.184:6809/82782","cluster_addr":"172.21.15.184:6811/82782","heartbeat_back_addr":"172.21.15.184:6815/82782","heartbeat_front_addr":"172.21.15.184:6813/82782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:08:14.873 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-29T22:08:14.874 INFO:tasks.ceph:Creating RBD pool 2023-09-29T22:08:14.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool create rbd 8 2023-09-29T22:08:15.226 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'rbd' created 2023-09-29T22:08:15.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T22:08:16.226 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T22:08:16.244 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-09-29T22:08:16.245 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-09-29T22:08:16.245 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-09-29T22:08:16.245 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-09-29T22:08:16.245 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T22:08:16.598 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:16.609 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":82445},{"type":"v1","addr":"172.21.15.184:6825","nonce":82445}]},"active_addr":"172.21.15.184:6825/82445","active_change":"2023-09-29T22:08:10.983773+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":3534227633}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":4203102620}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1425653938}]}]} 2023-09-29T22:08:16.613 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-09-29T22:08:16.613 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-09-29T22:08:16.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T22:08:16.956 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:16.956 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":10,"fsid":"713bf48e-c521-49ea-a8a1-6dd1a1ff10c6","created":"2023-09-29T22:08:06.750233+0000","modified":"2023-09-29T22:08:16.223830+0000","last_up_change":"2023-09-29T22:08:14.212208+0000","last_in_change":"2023-09-29T22:08:07.606219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T22:08:15.029158+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T22:08:15.168625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5916e3e-d36e-4088-b2e2-226a39dcc4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":82781},{"type":"v1","addr":"172.21.15.184:6801","nonce":82781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":82781},{"type":"v1","addr":"172.21.15.184:6803","nonce":82781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":82781},{"type":"v1","addr":"172.21.15.184:6807","nonce":82781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":82781},{"type":"v1","addr":"172.21.15.184:6805","nonce":82781}]},"public_addr":"172.21.15.184:6801/82781","cluster_addr":"172.21.15.184:6803/82781","heartbeat_back_addr":"172.21.15.184:6807/82781","heartbeat_front_addr":"172.21.15.184:6805/82781","state":["exists","up"]},{"osd":1,"uuid":"b2ec478a-203e-4598-a7d6-6e6687417b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":82783},{"type":"v1","addr":"172.21.15.184:6817","nonce":82783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":82783},{"type":"v1","addr":"172.21.15.184:6819","nonce":82783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":82783},{"type":"v1","addr":"172.21.15.184:6823","nonce":82783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":82783},{"type":"v1","addr":"172.21.15.184:6821","nonce":82783}]},"public_addr":"172.21.15.184:6817/82783","cluster_addr":"172.21.15.184:6819/82783","heartbeat_back_addr":"172.21.15.184:6823/82783","heartbeat_front_addr":"172.21.15.184:6821/82783","state":["exists","up"]},{"osd":2,"uuid":"d4d6c3a7-c7c3-428a-8695-7ba22e5af108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":82782},{"type":"v1","addr":"172.21.15.184:6809","nonce":82782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":82782},{"type":"v1","addr":"172.21.15.184:6811","nonce":82782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":82782},{"type":"v1","addr":"172.21.15.184:6815","nonce":82782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":82782},{"type":"v1","addr":"172.21.15.184:6813","nonce":82782}]},"public_addr":"172.21.15.184:6809/82782","cluster_addr":"172.21.15.184:6811/82782","heartbeat_back_addr":"172.21.15.184:6815/82782","heartbeat_front_addr":"172.21.15.184:6813/82782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:12.001774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:12.021378+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:11.952718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:08:16.968 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-09-29T22:08:16.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T22:08:17.244 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:17.244 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":11,"fsid":"713bf48e-c521-49ea-a8a1-6dd1a1ff10c6","created":"2023-09-29T22:08:06.750233+0000","modified":"2023-09-29T22:08:17.229441+0000","last_up_change":"2023-09-29T22:08:14.212208+0000","last_in_change":"2023-09-29T22:08:07.606219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T22:08:15.029158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T22:08:15.168625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5916e3e-d36e-4088-b2e2-226a39dcc4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":82781},{"type":"v1","addr":"172.21.15.184:6801","nonce":82781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":82781},{"type":"v1","addr":"172.21.15.184:6803","nonce":82781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":82781},{"type":"v1","addr":"172.21.15.184:6807","nonce":82781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":82781},{"type":"v1","addr":"172.21.15.184:6805","nonce":82781}]},"public_addr":"172.21.15.184:6801/82781","cluster_addr":"172.21.15.184:6803/82781","heartbeat_back_addr":"172.21.15.184:6807/82781","heartbeat_front_addr":"172.21.15.184:6805/82781","state":["exists","up"]},{"osd":1,"uuid":"b2ec478a-203e-4598-a7d6-6e6687417b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":82783},{"type":"v1","addr":"172.21.15.184:6817","nonce":82783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":82783},{"type":"v1","addr":"172.21.15.184:6819","nonce":82783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":82783},{"type":"v1","addr":"172.21.15.184:6823","nonce":82783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":82783},{"type":"v1","addr":"172.21.15.184:6821","nonce":82783}]},"public_addr":"172.21.15.184:6817/82783","cluster_addr":"172.21.15.184:6819/82783","heartbeat_back_addr":"172.21.15.184:6823/82783","heartbeat_front_addr":"172.21.15.184:6821/82783","state":["exists","up"]},{"osd":2,"uuid":"d4d6c3a7-c7c3-428a-8695-7ba22e5af108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":82782},{"type":"v1","addr":"172.21.15.184:6809","nonce":82782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":82782},{"type":"v1","addr":"172.21.15.184:6811","nonce":82782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":82782},{"type":"v1","addr":"172.21.15.184:6815","nonce":82782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":82782},{"type":"v1","addr":"172.21.15.184:6813","nonce":82782}]},"public_addr":"172.21.15.184:6809/82782","cluster_addr":"172.21.15.184:6811/82782","heartbeat_back_addr":"172.21.15.184:6815/82782","heartbeat_front_addr":"172.21.15.184:6813/82782","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:12.001774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:12.021378+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T22:08:11.952718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T22:08:17.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-09-29T22:08:17.257 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-09-29T22:08:17.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-09-29T22:08:17.397 INFO:teuthology.orchestra.run.smithi184.stdout:34359738373 2023-09-29T22:08:17.397 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T22:08:17.408 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2023-09-29T22:08:17.408 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T22:08:17.412 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2023-09-29T22:08:17.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T22:08:17.707 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2023-09-29T22:08:17.723 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2023-09-29T22:08:17.741 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2023-09-29T22:08:17.753 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2023-09-29T22:08:17.764 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2023-09-29T22:08:17.776 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2023-09-29T22:08:18.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T22:08:18.754 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T22:08:18.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T22:08:19.029 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2023-09-29T22:08:19.046 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2023-09-29T22:08:19.070 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2023-09-29T22:08:19.083 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2023-09-29T22:08:19.104 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2023-09-29T22:08:19.116 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2023-09-29T22:08:20.047 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T22:08:20.084 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T22:08:20.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T22:08:20.348 INFO:teuthology.orchestra.run.smithi184.stdout:34359738373 2023-09-29T22:08:20.361 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738373 for osd.1 2023-09-29T22:08:20.361 DEBUG:teuthology.parallel:result is None 2023-09-29T22:08:20.412 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2023-09-29T22:08:20.425 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2023-09-29T22:08:20.425 DEBUG:teuthology.parallel:result is None 2023-09-29T22:08:20.450 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2023-09-29T22:08:20.462 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2023-09-29T22:08:20.462 DEBUG:teuthology.parallel:result is None 2023-09-29T22:08:20.462 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-09-29T22:08:20.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-29T22:08:20.773 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:20.773 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2023-09-29T22:08:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-09-29T22:08:18.988194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82272,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490528,"statfs":{"total":322122547200,"available":322038300672,"internally_reserved":0,"allocated":1449984,"data_stored":1246254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231510+0000","last_change":"2023-09-29T22:08:16.234708+0000","last_active":"2023-09-29T22:08:17.231510+0000","last_peered":"2023-09-29T22:08:17.231510+0000","last_clean":"2023-09-29T22:08:17.231510+0000","last_became_active":"2023-09-29T22:08:16.234437+0000","last_became_peered":"2023-09-29T22:08:16.234437+0000","last_unstale":"2023-09-29T22:08:17.231510+0000","last_undegraded":"2023-09-29T22:08:17.231510+0000","last_fullsized":"2023-09-29T22:08:17.231510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:16:38.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231429+0000","last_change":"2023-09-29T22:08:16.235209+0000","last_active":"2023-09-29T22:08:17.231429+0000","last_peered":"2023-09-29T22:08:17.231429+0000","last_clean":"2023-09-29T22:08:17.231429+0000","last_became_active":"2023-09-29T22:08:16.234933+0000","last_became_peered":"2023-09-29T22:08:16.234933+0000","last_unstale":"2023-09-29T22:08:17.231429+0000","last_undegraded":"2023-09-29T22:08:17.231429+0000","last_fullsized":"2023-09-29T22:08:17.231429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:39:38.645030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231445+0000","last_change":"2023-09-29T22:08:16.234568+0000","last_active":"2023-09-29T22:08:17.231445+0000","last_peered":"2023-09-29T22:08:17.231445+0000","last_clean":"2023-09-29T22:08:17.231445+0000","last_became_active":"2023-09-29T22:08:16.234322+0000","last_became_peered":"2023-09-29T22:08:16.234322+0000","last_unstale":"2023-09-29T22:08:17.231445+0000","last_undegraded":"2023-09-29T22:08:17.231445+0000","last_fullsized":"2023-09-29T22:08:17.231445+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:51:51.234953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231436+0000","last_change":"2023-09-29T22:08:16.234819+0000","last_active":"2023-09-29T22:08:17.231436+0000","last_peered":"2023-09-29T22:08:17.231436+0000","last_clean":"2023-09-29T22:08:17.231436+0000","last_became_active":"2023-09-29T22:08:16.234556+0000","last_became_peered":"2023-09-29T22:08:16.234556+0000","last_unstale":"2023-09-29T22:08:17.231436+0000","last_undegraded":"2023-09-29T22:08:17.231436+0000","last_fullsized":"2023-09-29T22:08:17.231436+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:59:14.886264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:16.236472+0000","last_change":"2023-09-29T22:08:16.233169+0000","last_active":"2023-09-29T22:08:16.236472+0000","last_peered":"2023-09-29T22:08:16.236472+0000","last_clean":"2023-09-29T22:08:16.236472+0000","last_became_active":"2023-09-29T22:08:16.232912+0000","last_became_peered":"2023-09-29T22:08:16.232912+0000","last_unstale":"2023-09-29T22:08:16.236472+0000","last_undegraded":"2023-09-29T22:08:16.236472+0000","last_fullsized":"2023-09-29T22:08:16.236472+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:53.399611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:16.235677+0000","last_change":"2023-09-29T22:08:16.233741+0000","last_active":"2023-09-29T22:08:16.235677+0000","last_peered":"2023-09-29T22:08:16.235677+0000","last_clean":"2023-09-29T22:08:16.235677+0000","last_became_active":"2023-09-29T22:08:16.233498+0000","last_became_peered":"2023-09-29T22:08:16.233498+0000","last_unstale":"2023-09-29T22:08:16.235677+0000","last_undegraded":"2023-09-29T22:08:16.235677+0000","last_fullsized":"2023-09-29T22:08:16.235677+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:28:47.738432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:16.238280+0000","last_change":"2023-09-29T22:08:16.233840+0000","last_active":"2023-09-29T22:08:16.238280+0000","last_peered":"2023-09-29T22:08:16.238280+0000","last_clean":"2023-09-29T22:08:16.238280+0000","last_became_active":"2023-09-29T22:08:16.233615+0000","last_became_peered":"2023-09-29T22:08:16.233615+0000","last_unstale":"2023-09-29T22:08:16.238280+0000","last_undegraded":"2023-09-29T22:08:16.238280+0000","last_fullsized":"2023-09-29T22:08:16.238280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:49:17.741994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231481+0000","last_change":"2023-09-29T22:08:16.233516+0000","last_active":"2023-09-29T22:08:17.231481+0000","last_peered":"2023-09-29T22:08:17.231481+0000","last_clean":"2023-09-29T22:08:17.231481+0000","last_became_active":"2023-09-29T22:08:16.233273+0000","last_became_peered":"2023-09-29T22:08:16.233273+0000","last_unstale":"2023-09-29T22:08:17.231481+0000","last_undegraded":"2023-09-29T22:08:17.231481+0000","last_fullsized":"2023-09-29T22:08:17.231481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:01.493625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231454+0000","last_change":"2023-09-29T22:08:16.235709+0000","last_active":"2023-09-29T22:08:17.231454+0000","last_peered":"2023-09-29T22:08:17.231454+0000","last_clean":"2023-09-29T22:08:17.231454+0000","last_became_active":"2023-09-29T22:08:16.234856+0000","last_became_peered":"2023-09-29T22:08:16.234856+0000","last_unstale":"2023-09-29T22:08:17.231454+0000","last_undegraded":"2023-09-29T22:08:17.231454+0000","last_fullsized":"2023-09-29T22:08:17.231454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:54.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27620,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829980,"statfs":{"total":107374182400,"available":107345899520,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27616,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829984,"statfs":{"total":107374182400,"available":107345903616,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T22:08:20.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-29T22:08:21.058 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:21.059 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2023-09-29T22:08:21.070 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-09-29T22:08:20.988489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82288,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490512,"statfs":{"total":322122547200,"available":322038284288,"internally_reserved":0,"allocated":1466368,"data_stored":1253950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000295"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231510+0000","last_change":"2023-09-29T22:08:16.234708+0000","last_active":"2023-09-29T22:08:17.231510+0000","last_peered":"2023-09-29T22:08:17.231510+0000","last_clean":"2023-09-29T22:08:17.231510+0000","last_became_active":"2023-09-29T22:08:16.234437+0000","last_became_peered":"2023-09-29T22:08:16.234437+0000","last_unstale":"2023-09-29T22:08:17.231510+0000","last_undegraded":"2023-09-29T22:08:17.231510+0000","last_fullsized":"2023-09-29T22:08:17.231510+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:16:38.639429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231429+0000","last_change":"2023-09-29T22:08:16.235209+0000","last_active":"2023-09-29T22:08:17.231429+0000","last_peered":"2023-09-29T22:08:17.231429+0000","last_clean":"2023-09-29T22:08:17.231429+0000","last_became_active":"2023-09-29T22:08:16.234933+0000","last_became_peered":"2023-09-29T22:08:16.234933+0000","last_unstale":"2023-09-29T22:08:17.231429+0000","last_undegraded":"2023-09-29T22:08:17.231429+0000","last_fullsized":"2023-09-29T22:08:17.231429+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:39:38.645030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231445+0000","last_change":"2023-09-29T22:08:16.234568+0000","last_active":"2023-09-29T22:08:17.231445+0000","last_peered":"2023-09-29T22:08:17.231445+0000","last_clean":"2023-09-29T22:08:17.231445+0000","last_became_active":"2023-09-29T22:08:16.234322+0000","last_became_peered":"2023-09-29T22:08:16.234322+0000","last_unstale":"2023-09-29T22:08:17.231445+0000","last_undegraded":"2023-09-29T22:08:17.231445+0000","last_fullsized":"2023-09-29T22:08:17.231445+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:51:51.234953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231436+0000","last_change":"2023-09-29T22:08:16.234819+0000","last_active":"2023-09-29T22:08:17.231436+0000","last_peered":"2023-09-29T22:08:17.231436+0000","last_clean":"2023-09-29T22:08:17.231436+0000","last_became_active":"2023-09-29T22:08:16.234556+0000","last_became_peered":"2023-09-29T22:08:16.234556+0000","last_unstale":"2023-09-29T22:08:17.231436+0000","last_undegraded":"2023-09-29T22:08:17.231436+0000","last_fullsized":"2023-09-29T22:08:17.231436+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:59:14.886264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.396907+0000","last_change":"2023-09-29T22:08:16.233169+0000","last_active":"2023-09-29T22:08:17.396907+0000","last_peered":"2023-09-29T22:08:17.396907+0000","last_clean":"2023-09-29T22:08:17.396907+0000","last_became_active":"2023-09-29T22:08:16.232912+0000","last_became_peered":"2023-09-29T22:08:16.232912+0000","last_unstale":"2023-09-29T22:08:17.396907+0000","last_undegraded":"2023-09-29T22:08:17.396907+0000","last_fullsized":"2023-09-29T22:08:17.396907+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:53.399611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:16.235677+0000","last_change":"2023-09-29T22:08:16.233741+0000","last_active":"2023-09-29T22:08:16.235677+0000","last_peered":"2023-09-29T22:08:16.235677+0000","last_clean":"2023-09-29T22:08:16.235677+0000","last_became_active":"2023-09-29T22:08:16.233498+0000","last_became_peered":"2023-09-29T22:08:16.233498+0000","last_unstale":"2023-09-29T22:08:16.235677+0000","last_undegraded":"2023-09-29T22:08:16.235677+0000","last_fullsized":"2023-09-29T22:08:16.235677+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:28:47.738432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-29T22:08:16.238280+0000","last_change":"2023-09-29T22:08:16.233840+0000","last_active":"2023-09-29T22:08:16.238280+0000","last_peered":"2023-09-29T22:08:16.238280+0000","last_clean":"2023-09-29T22:08:16.238280+0000","last_became_active":"2023-09-29T22:08:16.233615+0000","last_became_peered":"2023-09-29T22:08:16.233615+0000","last_unstale":"2023-09-29T22:08:16.238280+0000","last_undegraded":"2023-09-29T22:08:16.238280+0000","last_fullsized":"2023-09-29T22:08:16.238280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:49:17.741994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231481+0000","last_change":"2023-09-29T22:08:16.233516+0000","last_active":"2023-09-29T22:08:17.231481+0000","last_peered":"2023-09-29T22:08:17.231481+0000","last_clean":"2023-09-29T22:08:17.231481+0000","last_became_active":"2023-09-29T22:08:16.233273+0000","last_became_peered":"2023-09-29T22:08:16.233273+0000","last_unstale":"2023-09-29T22:08:17.231481+0000","last_undegraded":"2023-09-29T22:08:17.231481+0000","last_fullsized":"2023-09-29T22:08:17.231481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:01.493625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T22:08:17.231454+0000","last_change":"2023-09-29T22:08:16.235709+0000","last_active":"2023-09-29T22:08:17.231454+0000","last_peered":"2023-09-29T22:08:17.231454+0000","last_clean":"2023-09-29T22:08:17.231454+0000","last_became_active":"2023-09-29T22:08:16.234856+0000","last_became_peered":"2023-09-29T22:08:16.234856+0000","last_unstale":"2023-09-29T22:08:17.231454+0000","last_undegraded":"2023-09-29T22:08:17.231454+0000","last_fullsized":"2023-09-29T22:08:17.231454+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T22:08:15.215999+0000","last_clean_scrub_stamp":"2023-09-29T22:08:15.215999+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:54.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":688128,"data_stored":615685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T22:08:21.071 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-09-29T22:08:21.071 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-09-29T22:08:21.072 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-09-29T22:08:21.072 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-09-29T22:08:21.385 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:08:21.386 INFO:teuthology.orchestra.run.smithi184.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-29T22:08:21.398 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-09-29T22:08:21.399 INFO:teuthology.run_tasks:Running task rgw... 2023-09-29T22:08:21.417 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-09-29T22:08:21.417 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-09-29T22:08:21.417 INFO:tasks.rgw:Creating data pools 2023-09-29T22:08:21.417 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-09-29T22:08:21.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-29T22:08:21.915 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' created 2023-09-29T22:08:21.936 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-29T22:08:22.924 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T22:08:22.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-29T22:08:23.932 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' created 2023-09-29T22:08:23.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-29T22:08:24.936 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T22:08:24.960 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-09-29T22:08:24.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-29T22:08:25.236 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-29T22:08:25.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-29T22:08:25.936 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T22:08:25.949 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-29T22:08:26.227 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-29T22:08:26.240 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-29T22:08:26.939 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T22:08:26.952 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-09-29T22:08:26.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-29T22:08:27.787 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' created 2023-09-29T22:08:27.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-29T22:08:28.790 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T22:08:28.823 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-29T22:08:29.795 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' created 2023-09-29T22:08:29.818 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-29T22:08:30.801 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T22:08:30.832 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-09-29T22:08:30.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-29T22:08:31.112 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-29T22:08:31.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-29T22:08:31.800 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T22:08:31.814 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-29T22:08:32.090 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-29T22:08:32.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-29T22:08:32.802 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T22:08:32.816 DEBUG:tasks.rgw:Pools created 2023-09-29T22:08:32.816 INFO:tasks.rgw:Configuring compression type = random 2023-09-29T22:08:32.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-09-29T22:08:32.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-09-29T22:08:32.817 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-09-29T22:08:32.922 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:32.922 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:32.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:32.935+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.022+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.022+0000 7f97a5329880 20 realm 2023-09-29T22:08:34.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.022+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.024 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.025+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.025+0000 7f97a5329880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T22:08:34.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.025+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.025+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.025+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.028+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.029+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.029+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.029+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.029+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.029+0000 7f97a5329880 10 Creating default zonegroup 2023-09-29T22:08:34.030 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.030+0000 7f97a5329880 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-29T22:08:34.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.040+0000 7f97a5329880 10 WARNING: driver->list_zones() returned r=-2 2023-09-29T22:08:34.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.040+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.040+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.042+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.043+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.043+0000 7f97a5329880 10 could not read realm id: (2) No such file or directory 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.043+0000 7f97a5329880 10 WARNING: failed to set zone as default, r=-22 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.043+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.043+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.045+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.045+0000 7f97a5329880 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T22:08:34.045 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.045+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.047+0000 7f97a5329880 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T22:08:34.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.047+0000 7f97a5329880 10 Using default name default 2023-09-29T22:08:34.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.047+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:34.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 zone default found 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 4 Realm: () 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 4 ZoneGroup: default (923fb044-4c23-46b9-9a3f-4d875064fcb2) 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 4 Zone: default (069ae1d1-08b5-4c7e-9076-de3c3b623fc9) 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 zonegroup default 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:34.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:34.048+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:36.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:36.039+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:36.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:36.039+0000 7f97a5329880 20 rados->read ofs=0 len=0 2023-09-29T22:08:36.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:36.040+0000 7f97a5329880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:36.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:36.040+0000 7f97a5329880 20 started sync module instance, tier type = 2023-09-29T22:08:36.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:36.040+0000 7f97a5329880 20 started zone id=069ae1d1-08b5-4c7e-9076-de3c3b623fc9 (name=default) with tier type = 2023-09-29T22:08:38.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=0 2023-09-29T22:08:38.068 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=1 2023-09-29T22:08:38.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=2 2023-09-29T22:08:38.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=3 2023-09-29T22:08:38.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=4 2023-09-29T22:08:38.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=5 2023-09-29T22:08:38.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.068+0000 7f97a5329880 20 add_watcher() i=6 2023-09-29T22:08:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.069+0000 7f97a5329880 20 add_watcher() i=7 2023-09-29T22:08:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.069+0000 7f97a5329880 2 all 8 watchers are set, enabling cache 2023-09-29T22:08:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.069+0000 7f97a5329880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-29T22:08:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.069+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-29T22:08:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.069+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:38.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-29T22:08:38.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-29T22:08:38.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:38.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-29T22:08:38.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.070+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-29T22:08:38.074 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.072+0000 7f97a5329880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:38.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.076+0000 7f9788bce640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:38.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.076+0000 7f97a5329880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:38.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:38.076+0000 7f97a5329880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:41.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.047+0000 7f97a5329880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-29T22:08:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.047+0000 7f97a5329880 5 note: GC not initialized 2023-09-29T22:08:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.047+0000 7f97843c5640 20 reqs_thread_entry: start 2023-09-29T22:08:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.048+0000 7f97a5329880 20 init_complete bucket index max shards: 11 2023-09-29T22:08:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.048+0000 7f97a5329880 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T22:08:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.048+0000 7f97823c1640 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 22:09:11 2023 2023-09-29T22:08:41.050 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-09-29T22:08:41.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.048+0000 7f9781b9f640 20 reqs_thread_entry: start 2023-09-29T22:08:41.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.066+0000 7f9788bce640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:41.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.069+0000 7f97a5329880 20 remove_watcher() i=0 2023-09-29T22:08:41.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.069+0000 7f97a5329880 2 removed watcher, disabling cache 2023-09-29T22:08:41.072 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.071+0000 7f97a5329880 20 remove_watcher() i=1 2023-09-29T22:08:41.073 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.072+0000 7f97a5329880 20 remove_watcher() i=2 2023-09-29T22:08:41.076 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.075+0000 7f97a5329880 20 remove_watcher() i=3 2023-09-29T22:08:41.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.078+0000 7f97a5329880 20 remove_watcher() i=4 2023-09-29T22:08:41.079 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.079+0000 7f97a5329880 20 remove_watcher() i=5 2023-09-29T22:08:41.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.080+0000 7f97a5329880 20 remove_watcher() i=6 2023-09-29T22:08:41.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.082+0000 7f97a5329880 20 remove_watcher() i=7 2023-09-29T22:08:41.086 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2023-09-29T22:08:41.089 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T22:08:41.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:41.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:41.090 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T22:08:41.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:41.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:41.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.147+0000 7f2ee4b2f880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:41.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.147+0000 7f2ee4b2f880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:41.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.150+0000 7f2ec4cd4640 20 reqs_thread_entry: start 2023-09-29T22:08:41.157 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:41.157 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "069ae1d1-08b5-4c7e-9076-de3c3b623fc9", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T22:08:41.158 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:08:41.159 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.160 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:41.161 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2023-09-29T22:08:41.161 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T22:08:41.161 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:41.162 DEBUG:tasks.util.rgw: json result: {'id': '069ae1d1-08b5-4c7e-9076-de3c3b623fc9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T22:08:41.162 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-09-29T22:08:41.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-09-29T22:08:41.162 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-09-29T22:08:41.202 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:41.202 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:41.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.211+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 realm 2023-09-29T22:08:41.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 rados_obj.operate() r=0 bl.length=890 2023-09-29T22:08:41.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.212+0000 7f5332326880 20 searching for the correct realm 2023-09-29T22:08:41.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 RGWRados::pool_iterate: got zonegroup_info.923fb044-4c23-46b9-9a3f-4d875064fcb2 2023-09-29T22:08:41.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 RGWRados::pool_iterate: got zone_info.069ae1d1-08b5-4c7e-9076-de3c3b623fc9 2023-09-29T22:08:41.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 RGWRados::pool_iterate: got zone_names.default 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.219+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 zone default found 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 4 Realm: () 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 4 ZoneGroup: default (923fb044-4c23-46b9-9a3f-4d875064fcb2) 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 4 Zone: default (069ae1d1-08b5-4c7e-9076-de3c3b623fc9) 2023-09-29T22:08:41.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 zonegroup default 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados->read ofs=0 len=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 started sync module instance, tier type = 2023-09-29T22:08:41.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.220+0000 7f5332326880 20 started zone id=069ae1d1-08b5-4c7e-9076-de3c3b623fc9 (name=default) with tier type = 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=0 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=1 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=2 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=3 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=4 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=5 2023-09-29T22:08:41.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=6 2023-09-29T22:08:41.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 20 add_watcher() i=7 2023-09-29T22:08:41.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.243+0000 7f5332326880 2 all 8 watchers are set, enabling cache 2023-09-29T22:08:41.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.245+0000 7f5315bae640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:41.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.245+0000 7f5332326880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:41.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.245+0000 7f5332326880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.248+0000 7f5332326880 5 note: GC not initialized 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.249+0000 7f53113a5640 20 reqs_thread_entry: start 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.249+0000 7f5332326880 20 init_complete bucket index max shards: 11 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.249+0000 7f5332326880 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.249+0000 7f530f3a1640 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 22:09:11 2023 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr: 2023-09-29T22:08:41.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.249+0000 7f530eb7f640 20 reqs_thread_entry: start 2023-09-29T22:08:41.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.256+0000 7f5315bae640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:41.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.257+0000 7f5332326880 20 remove_watcher() i=0 2023-09-29T22:08:41.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.257+0000 7f5332326880 2 removed watcher, disabling cache 2023-09-29T22:08:41.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.258+0000 7f5332326880 20 remove_watcher() i=1 2023-09-29T22:08:41.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.259+0000 7f5332326880 20 remove_watcher() i=2 2023-09-29T22:08:41.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.261+0000 7f5332326880 20 remove_watcher() i=3 2023-09-29T22:08:41.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.261+0000 7f5332326880 20 remove_watcher() i=4 2023-09-29T22:08:41.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.262+0000 7f5332326880 20 remove_watcher() i=5 2023-09-29T22:08:41.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.263+0000 7f5332326880 20 remove_watcher() i=6 2023-09-29T22:08:41.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.264+0000 7f5332326880 20 remove_watcher() i=7 2023-09-29T22:08:41.267 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2023-09-29T22:08:41.269 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T22:08:41.269 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:41.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:41.269 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T22:08:41.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:41.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:41.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.363+0000 7f68c3c13880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:41.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.363+0000 7f68c3c13880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:41.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:41.365+0000 7f68a38c0640 20 reqs_thread_entry: start 2023-09-29T22:08:41.372 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:41.372 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "069ae1d1-08b5-4c7e-9076-de3c3b623fc9", 2023-09-29T22:08:41.372 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2023-09-29T22:08:41.372 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T22:08:41.372 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T22:08:41.373 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:08:41.374 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:41.375 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:41.376 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2023-09-29T22:08:41.376 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T22:08:41.376 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:41.377 DEBUG:tasks.util.rgw: json result: {'id': '069ae1d1-08b5-4c7e-9076-de3c3b623fc9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T22:08:41.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-09-29T22:08:41.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-09-29T22:08:41.377 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-09-29T22:08:41.485 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:41.486 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:41.497 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:41.497+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.830+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.830+0000 7f6068632880 20 realm 2023-09-29T22:08:42.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.830+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.831+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.832+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.832+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.832+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.832+0000 7f6068632880 10 Creating default zonegroup 2023-09-29T22:08:42.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.833+0000 7f6068632880 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-29T22:08:42.844 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.844+0000 7f6068632880 10 WARNING: driver->list_zones() returned r=-2 2023-09-29T22:08:42.844 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.844+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.844 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.844+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.846 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.846 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.847 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 10 could not read realm id: (2) No such file or directory 2023-09-29T22:08:42.847 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 10 WARNING: failed to set zone as default, r=-22 2023-09-29T22:08:42.847 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.847 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.846+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.848 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.848+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.849 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.848+0000 7f6068632880 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T22:08:42.849 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.848+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.852 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.851+0000 7f6068632880 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T22:08:42.852 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.851+0000 7f6068632880 10 Using default name default 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.851+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 zone default found 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 4 Realm: () 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 4 ZoneGroup: default (e96a78ed-4ff7-4361-b0c2-f717fc5bedc5) 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 4 Zone: default (a17331b2-22cf-4f3a-9bb8-349fbcdcbd32) 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 zonegroup default 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:42.853 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:42.854 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:42.852+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:44.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:44.840+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:44.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:44.840+0000 7f6068632880 20 rados->read ofs=0 len=0 2023-09-29T22:08:44.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:44.841+0000 7f6068632880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:44.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:44.841+0000 7f6068632880 20 started sync module instance, tier type = 2023-09-29T22:08:44.842 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:44.841+0000 7f6068632880 20 started zone id=a17331b2-22cf-4f3a-9bb8-349fbcdcbd32 (name=default) with tier type = 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=0 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=1 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=2 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=3 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=4 2023-09-29T22:08:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=5 2023-09-29T22:08:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.871+0000 7f6068632880 20 add_watcher() i=6 2023-09-29T22:08:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 add_watcher() i=7 2023-09-29T22:08:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 2 all 8 watchers are set, enabling cache 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-29T22:08:46.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.874+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-29T22:08:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.875+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-29T22:08:46.877 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.876+0000 7f6068632880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T22:08:46.884 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.884+0000 7f604b75f640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:46.885 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.884+0000 7f6068632880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:46.885 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:46.884+0000 7f6068632880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:49.857 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.856+0000 7f6068632880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-29T22:08:49.857 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.856+0000 7f6068632880 5 note: GC not initialized 2023-09-29T22:08:49.857 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.857+0000 7f6046f56640 20 reqs_thread_entry: start 2023-09-29T22:08:49.858 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.857+0000 7f6068632880 20 init_complete bucket index max shards: 11 2023-09-29T22:08:49.858 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.857+0000 7f6068632880 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T22:08:49.858 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.857+0000 7f6044f52640 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 22:09:20 2023 2023-09-29T22:08:49.858 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-09-29T22:08:49.858 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.858+0000 7f6044730640 20 reqs_thread_entry: start 2023-09-29T22:08:49.878 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.877+0000 7f604b75f640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:49.880 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.880+0000 7f6068632880 20 remove_watcher() i=0 2023-09-29T22:08:49.880 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.880+0000 7f6068632880 2 removed watcher, disabling cache 2023-09-29T22:08:49.881 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.881+0000 7f6068632880 20 remove_watcher() i=1 2023-09-29T22:08:49.882 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.882+0000 7f6068632880 20 remove_watcher() i=2 2023-09-29T22:08:49.883 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.883+0000 7f6068632880 20 remove_watcher() i=3 2023-09-29T22:08:49.884 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.884+0000 7f6068632880 20 remove_watcher() i=4 2023-09-29T22:08:49.886 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.886+0000 7f6068632880 20 remove_watcher() i=5 2023-09-29T22:08:49.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.887+0000 7f6068632880 20 remove_watcher() i=6 2023-09-29T22:08:49.888 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:49.888+0000 7f6068632880 20 remove_watcher() i=7 2023-09-29T22:08:49.973 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2023-09-29T22:08:49.976 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T22:08:49.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:49.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:49.977 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T22:08:50.030 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:50.030 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:50.040 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.040+0000 7f6551219880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:50.040 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.040+0000 7f6551219880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:50.043 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.043+0000 7f6530ec7640 20 reqs_thread_entry: start 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a17331b2-22cf-4f3a-9bb8-349fbcdcbd32", 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2023-09-29T22:08:50.051 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:08:50.052 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T22:08:50.054 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:08:50.055 DEBUG:tasks.util.rgw: json result: {'id': 'a17331b2-22cf-4f3a-9bb8-349fbcdcbd32', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T22:08:50.055 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-09-29T22:08:50.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-09-29T22:08:50.055 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-09-29T22:08:50.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:50.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:50.107 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.107+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.107+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.107+0000 7f458bd37880 20 realm 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.107+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.110 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.110 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:50.110 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.110 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 rados_obj.operate() r=0 bl.length=890 2023-09-29T22:08:50.110 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.108+0000 7f458bd37880 20 searching for the correct realm 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 RGWRados::pool_iterate: got zonegroup_info.e96a78ed-4ff7-4361-b0c2-f717fc5bedc5 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 RGWRados::pool_iterate: got zone_info.a17331b2-22cf-4f3a-9bb8-349fbcdcbd32 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 RGWRados::pool_iterate: got zone_names.default 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T22:08:50.117 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 zone default found 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 4 Realm: () 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 4 ZoneGroup: default (e96a78ed-4ff7-4361-b0c2-f717fc5bedc5) 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 4 Zone: default (a17331b2-22cf-4f3a-9bb8-349fbcdcbd32) 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 zonegroup default 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.116+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.118 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 rados->read ofs=0 len=0 2023-09-29T22:08:50.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T22:08:50.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 started sync module instance, tier type = 2023-09-29T22:08:50.119 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.117+0000 7f458bd37880 20 started zone id=a17331b2-22cf-4f3a-9bb8-349fbcdcbd32 (name=default) with tier type = 2023-09-29T22:08:50.142 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=0 2023-09-29T22:08:50.142 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=1 2023-09-29T22:08:50.142 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=2 2023-09-29T22:08:50.142 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=3 2023-09-29T22:08:50.143 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=4 2023-09-29T22:08:50.143 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=5 2023-09-29T22:08:50.143 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.142+0000 7f458bd37880 20 add_watcher() i=6 2023-09-29T22:08:50.145 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.144+0000 7f458bd37880 20 add_watcher() i=7 2023-09-29T22:08:50.145 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.144+0000 7f458bd37880 2 all 8 watchers are set, enabling cache 2023-09-29T22:08:50.147 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.147+0000 7f456f6e9640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:50.147 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.147+0000 7f458bd37880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:50.147 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.147+0000 7f458bd37880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:50.151 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f458bd37880 5 note: GC not initialized 2023-09-29T22:08:50.151 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f456aee0640 20 reqs_thread_entry: start 2023-09-29T22:08:50.151 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f458bd37880 20 init_complete bucket index max shards: 11 2023-09-29T22:08:50.151 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f458bd37880 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T22:08:50.152 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f4568edc640 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 22:09:20 2023 2023-09-29T22:08:50.152 INFO:teuthology.orchestra.run.smithi184.stderr: 2023-09-29T22:08:50.152 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.151+0000 7f45686ba640 20 reqs_thread_entry: start 2023-09-29T22:08:50.158 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.158+0000 7f456f6e9640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T22:08:50.162 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.162+0000 7f458bd37880 20 remove_watcher() i=0 2023-09-29T22:08:50.162 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.162+0000 7f458bd37880 2 removed watcher, disabling cache 2023-09-29T22:08:50.164 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.164+0000 7f458bd37880 20 remove_watcher() i=1 2023-09-29T22:08:50.167 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.167+0000 7f458bd37880 20 remove_watcher() i=2 2023-09-29T22:08:50.170 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.170+0000 7f458bd37880 20 remove_watcher() i=3 2023-09-29T22:08:50.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.175+0000 7f458bd37880 20 remove_watcher() i=4 2023-09-29T22:08:50.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.176+0000 7f458bd37880 20 remove_watcher() i=5 2023-09-29T22:08:50.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.179+0000 7f458bd37880 20 remove_watcher() i=6 2023-09-29T22:08:50.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.179+0000 7f458bd37880 20 remove_watcher() i=7 2023-09-29T22:08:50.182 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2023-09-29T22:08:50.184 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T22:08:50.184 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:50.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T22:08:50.185 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T22:08:50.270 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:50.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:50.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.280+0000 7efe4f610880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T22:08:50.280 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.280+0000 7efe4f610880 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T22:08:50.282 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:08:50.282+0000 7efe2f332640 20 reqs_thread_entry: start 2023-09-29T22:08:50.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a17331b2-22cf-4f3a-9bb8-349fbcdcbd32", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T22:08:50.291 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:08:50.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:08:50.294 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:08:50.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2023-09-29T22:08:50.294 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T22:08:50.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:08:50.295 DEBUG:tasks.util.rgw: json result: {'id': 'a17331b2-22cf-4f3a-9bb8-349fbcdcbd32', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T22:08:50.295 INFO:tasks.rgw:Starting rgw... 2023-09-29T22:08:50.295 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-09-29T22:08:50.295 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T22:08:50.295 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-29T22:08:50.295 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-29T22:08:50.298 INFO:tasks.rgw.c1.client.0:Started 2023-09-29T22:08:50.298 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-09-29T22:08:50.298 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T22:08:50.299 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:08:50.299 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:08:50.301 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:08:50.301 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-09-29T22:08:50.301 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T22:08:50.301 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T22:08:50.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T22:08:50.303 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T22:08:50.303 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-09-29T22:08:50.303 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T22:08:50.304 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-09-29T22:08:50.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-09-29T22:08:50.305 INFO:tasks.rgw.c2.client.1:Started 2023-09-29T22:08:50.305 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8000/ 2023-09-29T22:08:50.306 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000/ 2023-09-29T22:08:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:08:50.365 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:08:50.366 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:08:50.366 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:08:50.366 INFO:teuthology.orchestra.run.smithi175.stderr:curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T22:08:51.367 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000/ 2023-09-29T22:08:51.385 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:08:51.385 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:08:51.386 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:08:51.387 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2023-09-29T22:08:51.387 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8001/ 2023-09-29T22:08:51.387 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001/ 2023-09-29T22:08:51.445 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:08:51.445 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:08:51.445 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:08:51.446 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2023-09-29T22:08:51.446 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8000/ 2023-09-29T22:08:51.447 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000/ 2023-09-29T22:08:51.480 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:08:51.481 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:08:51.481 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:08:51.482 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-29T22:08:51.483 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8001/ 2023-09-29T22:08:51.483 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001/ 2023-09-29T22:08:51.541 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:08:51.541 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:08:51.542 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:08:51.543 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-29T22:08:51.543 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-09-29T22:08:51.601 INFO:tasks.rgw_multisite:creating realm.. 2023-09-29T22:08:51.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.602 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:08:51.643 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:51.644 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-realm", 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout: "current_period": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 1 2023-09-29T22:08:51.671 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:51.672 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-09-29T22:08:51.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.673 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:08:51.756 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:51.757 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:51.770 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:51.769+0000 7fcb2a136880 -1 old format 2023-09-29T22:08:51.778 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:51.778 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "", 2023-09-29T22:08:51.779 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [], 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [] 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:08:51.780 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:08:51.781 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:51.782 INFO:tasks.rgw_multisite:creating master zone.. 2023-09-29T22:08:51.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:51.783 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --endpoints http://smithi175.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:08:51.874 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:51.874 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:51.889 INFO:teuthology.orchestra.run.smithi175.stderr:NOTICE: set zone's realm_id=de44a8c3-8a59-46e4-824e-eaf8e5fcea33 2023-09-29T22:08:51.910 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone1.rgw.meta:root", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone1.rgw.control", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone1.rgw.log", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2023-09-29T22:08:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone1.rgw.otp", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:08:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:51.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:08:51.914 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2023-09-29T22:08:51.914 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:51.915 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2023-09-29T22:08:53.059 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'test-zone1.rgw.buckets.data' created 2023-09-29T22:08:53.087 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2023-09-29T22:08:54.062 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2023-09-29T22:08:54.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:54.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:54.098 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:08:54.147 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:54.147 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone1.rgw.meta:root", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone1.rgw.control", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2023-09-29T22:08:54.168 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone1.rgw.log", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone1.rgw.otp", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:08:54.169 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:54.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2023-09-29T22:08:54.171 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:08:54.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:54.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:08:54.172 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:08:54.216 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:54.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:54.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:08:54.228+0000 7fe6fa815880 0 period (2bdb37d0-ae81-48f2-8aab-f47771c96a40 does not have zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e configured 2023-09-29T22:09:01.176 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:01.176 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:01.176 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 1, 2023-09-29T22:09:01.176 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:01.177 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:01.178 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": false, 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:01.179 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:01.180 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.182 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:01.183 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:01.184 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-29T22:09:01.185 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:09:01.186 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:01.187 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-29T22:09:01.187 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:09:01.188 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:09:01.188+0000 7fd99d9c5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 87524) UID: 0 2023-09-29T22:09:01.188 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:09:01.188+0000 7fd9a3a34880 -1 shutting down 2023-09-29T22:09:03.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-09-29T22:09:07.289 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-29T22:09:07.289 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-29T22:09:07.289 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-29T22:09:07.292 INFO:tasks.rgw.c1.client.0:Started 2023-09-29T22:09:07.293 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2023-09-29T22:09:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:09:07.314 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:07.314 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:07.314 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:09:07.314 INFO:teuthology.orchestra.run.smithi175.stderr:curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T22:09:08.316 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2023-09-29T22:09:08.336 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:08.337 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:08.338 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:09:08.339 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2023-09-29T22:09:08.339 INFO:tasks.rgw_multisite:creating admin user.. 2023-09-29T22:09:08.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.339 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:08.437 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:08.437 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "user_id": "realm-admin", 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "display_name": "Realm Admin", 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "email": "", 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "suspended": 0, 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "max_buckets": 1000, 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "subusers": [], 2023-09-29T22:09:08.468 INFO:teuthology.orchestra.run.smithi175.stdout: "keys": [ 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "realm-admin", 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "swift_keys": [], 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "caps": [], 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "op_mask": "read, write, delete", 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "system": true, 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2023-09-29T22:09:08.469 INFO:teuthology.orchestra.run.smithi175.stdout: "default_storage_class": "", 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_tags": [], 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:08.470 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: "temp_url_keys": [], 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "rgw", 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: "mfa_ids": [] 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:08.471 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:09:08.480 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-09-29T22:09:08.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.480 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi175.front.sepia.ceph.com:8000 --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:08.534 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:08.535 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:08.569 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:08.570 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:08.570 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-realm", 2023-09-29T22:09:08.570 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:08.570 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2 2023-09-29T22:09:08.570 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:08.571 INFO:tasks.rgw_multisite:creating zone.. 2023-09-29T22:09:08.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:08.573 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --endpoints http://smithi184.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:08.624 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:08.625 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:08.639 INFO:teuthology.orchestra.run.smithi184.stderr:NOTICE: set zone's realm_id=de44a8c3-8a59-46e4-824e-eaf8e5fcea33 2023-09-29T22:09:08.653 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2023-09-29T22:09:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:09:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2023-09-29T22:09:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2023-09-29T22:09:08.657 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:08.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2023-09-29T22:09:09.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-09-29T22:09:09.879 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'test-zone2.rgw.buckets.data' created 2023-09-29T22:09:09.906 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2023-09-29T22:09:10.880 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2023-09-29T22:09:10.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:10.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:10.913 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:10.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:10.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:10.983 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2023-09-29T22:09:10.984 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:10.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:09:10.986 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2023-09-29T22:09:10.987 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:10.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:10.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:10.988 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:11.037 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:11.037 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:11.049 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:11.049+0000 7fb08ef36880 0 period (6a43d71a-d207-46f3-bdd8-e7a78954a184 does not have zone e97e2b17-58fb-4667-bf95-8d29031e4378 configured 2023-09-29T22:09:17.959 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2, 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2023-09-29T22:09:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": true, 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2023-09-29T22:09:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:09:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:09:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled_features": [ 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:17.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2464975307 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1018467004 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2023-09-29T22:09:17.998 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:09:17.999 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "user_ratelimit": { 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_ratelimit": { 2023-09-29T22:09:18.000 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:09:18.001 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "anonymous_ratelimit": { 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:09:18.012 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:18.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:18.014 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-29T22:09:18.015 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:09:18.015 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T22:09:18.015+0000 7f06eaac0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 87421) UID: 0 2023-09-29T22:09:18.015 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T22:09:18.015+0000 7f06f0c32880 -1 shutting down 2023-09-29T22:09:23.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-29T22:09:24.116 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-29T22:09:24.116 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T22:09:24.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T22:09:24.119 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T22:09:24.119 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T22:09:24.140 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:24.141 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:24.141 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:09:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:09:24.142 INFO:teuthology.orchestra.run.smithi184.stderr:curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T22:09:25.143 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T22:09:25.162 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:25.162 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:25.163 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:09:25.163 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-29T22:09:25.164 INFO:tasks.rgw_multisite:creating zone.. 2023-09-29T22:09:25.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:25.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:25.164 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --endpoints http://smithi175.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:25.238 INFO:teuthology.orchestra.run.smithi175.stderr:NOTICE: set zone's realm_id=de44a8c3-8a59-46e4-824e-eaf8e5fcea33 2023-09-29T22:09:25.253 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:25.254 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:09:25.254 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:09:25.254 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone3.rgw.meta:root", 2023-09-29T22:09:25.254 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone3.rgw.control", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone3.rgw.log", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2023-09-29T22:09:25.255 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone3.rgw.otp", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:09:25.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:09:25.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:25.258 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:25.258 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:25.258 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:25.258 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2023-09-29T22:09:25.258 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:25.259 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2023-09-29T22:09:25.892 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'test-zone3.rgw.buckets.data' created 2023-09-29T22:09:25.931 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2023-09-29T22:09:26.892 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2023-09-29T22:09:26.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:26.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:26.923 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:26.976 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:26.977 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:26.995 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:26.995 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:09:26.995 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:09:26.995 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone3.rgw.meta:root", 2023-09-29T22:09:26.995 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone3.rgw.control", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone3.rgw.log", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2023-09-29T22:09:26.996 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone3.rgw.otp", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:09:26.997 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2023-09-29T22:09:26.998 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2023-09-29T22:09:26.999 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:27.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:27.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:27.000 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:27.045 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:27.058 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:09:27.058+0000 7f0fb451f880 0 period (6a43d71a-d207-46f3-bdd8-e7a78954a184 does not have zone 4e047d8a-7fd9-483a-b356-0bddc8ecae27 configured 2023-09-29T22:09:29.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-29T22:09:33.961 INFO:teuthology.orchestra.run.smithi175.stderr:Sending period to new master zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 3, 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-29T22:09:33.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:33.981 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:09:33.982 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:33.983 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:09:33.984 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:09:33.985 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:09:33.987 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.988 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:09:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-29T22:09:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:33.989 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:09:33.991 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:33.992 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.993 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:09:33.994 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:09:33.995 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:33.996 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:09:33.996 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:09:33.997 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-29T22:09:33.997+0000 7ffaaad9f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 87527) UID: 0 2023-09-29T22:09:33.997 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-29T22:09:33.997+0000 7ffab0724880 -1 shutting down 2023-09-29T22:09:37.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:09:40.099 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:09:40.099 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:09:40.099 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:09:40.102 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:09:40.103 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2023-09-29T22:09:40.126 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:40.126 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:09:40.128 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:09:40.128 INFO:teuthology.orchestra.run.smithi175.stderr:curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:09:41.129 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2023-09-29T22:09:41.148 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:09:41.148 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:09:41.149 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:09:41.150 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2023-09-29T22:09:41.150 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-09-29T22:09:41.150 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-09-29T22:09:41.162 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi 2023-09-29T22:09:41.187 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-09-29T22:09:41.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:41.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:41.188 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:41.243 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:41.281 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:09:41.281 INFO:teuthology.orchestra.run.smithi175.stdout: "user_id": "rgw-multisite-test-user", 2023-09-29T22:09:41.281 INFO:teuthology.orchestra.run.smithi175.stdout: "display_name": "Multisite Test User", 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "email": "", 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "suspended": 0, 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "max_buckets": 1000, 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "subusers": [], 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "keys": [ 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "rgw-multisite-test-user", 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "PTXB0TDQIX657E2O5ERJ", 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "XucwljtXMQCaAVuvqHwr40ZI7jU9XJvZBebT3ICo" 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:41.282 INFO:teuthology.orchestra.run.smithi175.stdout: "swift_keys": [], 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "caps": [ 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "roles", 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "perm": "*" 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "op_mask": "read, write, delete", 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "default_storage_class": "", 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_tags": [], 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:09:41.283 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:09:41.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:41.285 INFO:teuthology.orchestra.run.smithi175.stdout: "temp_url_keys": [], 2023-09-29T22:09:41.285 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "rgw", 2023-09-29T22:09:41.285 INFO:teuthology.orchestra.run.smithi175.stdout: "mfa_ids": [] 2023-09-29T22:09:41.285 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:09:41.285 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:09:41.292 INFO:rgw_multi.tests:meta checkpoint 2023-09-29T22:09:41.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:41.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:41.293 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:41.337 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:41.337 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:41.365 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:09:41.376 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:09:41.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.376 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:09:41.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:09:41.378 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:09:41.378 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:09:41.378 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:09:41.378 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:09:41.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:09:41.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:41.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:41.379 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:41.432 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:41.432 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:41.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:41.464+0000 7f5876a13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:41.465 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:41.465+0000 7f5876a13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:41.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:41.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:41.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:41.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:41.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:09:41Z" 2023-09-29T22:09:41.482 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:41.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:09:43.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:09:46.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:46.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:46.485 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:46.538 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:46.539 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:46.570 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:46.570+0000 7f5e19127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:46.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:46.571+0000 7f5e19127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.583 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:46.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:46.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:09:46.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:46.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:46.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:46.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:46.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:09:46Z" 2023-09-29T22:09:46.591 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:46.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:09:51.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:51.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:51.593 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:51.646 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:51.646 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:51.678 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:51.678+0000 7f3bc3d29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:51.679 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:51.679+0000 7f3bc3d29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:51.691 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:51.691 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:09:51.691 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:09:51.692 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:09:51.692 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:09:51.692 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:51.692 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:51.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:51.693 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:09:51.694 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:51.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:51.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:51.697 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:09:51Z" 2023-09-29T22:09:51.698 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:51.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:09:56.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:56.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:56.701 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:56.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:56.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:56.782 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:56.782+0000 7f83de322880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:56.783 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:09:56.783+0000 7f83de322880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:56.795 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:09:56.795 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:09:56.795 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:09:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:09:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:09:56Z" 2023-09-29T22:09:56.801 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:09:56.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:01.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:01.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:01.802 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:01.856 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:01.856 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:01.889 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:01.889+0000 7f7ca9329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:01.890 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:01.890+0000 7f7ca9329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:01.901 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:01.901 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:01.902 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:01.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:01.905 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:01.906 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:01.907 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:01Z" 2023-09-29T22:10:01.908 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:01.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:06.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:06.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:06.910 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:06.961 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:06.961 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:06.991 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:06.991+0000 7fa8fb012880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:06.992 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:06.992+0000 7fa8fb012880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:07.004 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:07.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:07.006 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:07.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:07.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:06Z" 2023-09-29T22:10:07.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:07.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:12.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:12.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:12.011 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:12.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:12.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:12.103 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:12.102+0000 7f5ac5337880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:12.103 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:12.103+0000 7f5ac5337880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:12.115 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:12Z" 2023-09-29T22:10:12.122 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:12.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:17.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:17.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:17.125 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:17.179 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:17.179 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:17.210 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:17.210+0000 7f62f6415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:17.211 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:17.211+0000 7f62f6415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:17.222 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:17.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:17.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:17.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:17.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:17.227 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:17.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:17.228 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:17Z" 2023-09-29T22:10:17.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:17.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:22.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:22.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:22.230 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:22.283 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:22.283 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:22.315 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:22.315+0000 7f94e5e99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:22.316 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:22.316+0000 7f94e5e99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:22.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:22.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:22.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:22.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:22.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:22Z" 2023-09-29T22:10:22.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:22.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:27.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:27.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:27.335 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:27.389 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:27.389 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:27.427 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:27.426+0000 7f733e215880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:27.427 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:27.427+0000 7f733e215880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:27.440 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:27.440 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:27.440 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:27.440 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:27.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:27.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:27.443 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:27.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:27Z" 2023-09-29T22:10:27.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:27.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:32.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:32.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:32.447 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:32.501 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:32.501 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:32.531 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:32.531+0000 7fb0e3536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:32.532 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:32.532+0000 7fb0e3536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:32.543 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:32.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:32.545 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:32.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:32.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.548 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:32.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:32.549 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:32.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:32.549 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:32Z" 2023-09-29T22:10:32.549 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:32.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:37.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:37.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:37.551 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:37.609 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:37.610 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:37.644 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:37.644+0000 7f8e25323880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:37.645 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:37.645+0000 7f8e25323880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:37.658 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:37.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:37.660 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:37.661 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:37.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.663 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:37.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:37.664 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:37.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:37.664 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:37Z" 2023-09-29T22:10:37.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:37.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:42.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:42.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:42.666 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:42.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:42.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:42.750 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:42.749+0000 7f0110c11880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:42.750 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:42.750+0000 7f0110c11880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:42.762 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:42.762 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:42.762 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:42.763 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:42.764 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:42.765 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:42.766 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:42.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:42.768 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:42Z" 2023-09-29T22:10:42.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:42.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:47.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:47.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:47.770 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:47.827 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:47.827 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:47.863 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:47.863+0000 7f1404a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:47.864 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:47.864+0000 7f1404a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:47.877 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:47.878 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:47.879 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:47.880 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:47.881 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:47.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:47Z" 2023-09-29T22:10:47.883 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:47.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:52.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:52.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:52.885 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:52.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:52.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:52.971 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:52.971+0000 7f6433030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:52.971 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:52.972+0000 7f6433030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:52.983 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:52.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:52.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:52.985 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:52.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:52.987 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:52.988 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:52.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:52Z" 2023-09-29T22:10:52.990 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:52.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:10:57.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:57.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:10:57.993 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:10:58.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:58.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:58.077 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:58.077+0000 7f50bf7a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:58.078 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:10:58.078+0000 7f50bf7a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:10:58.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:58.097 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:58.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:58.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:58.100 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:10:58.101 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:10:58.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.101 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:10:58.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:10:58.102 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:10:58.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:10:58.102 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:10:58Z" 2023-09-29T22:10:58.102 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:10:58.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:03.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:03.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:03.104 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:03.157 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:03.157 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:03.190 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:03.190+0000 7f8e7e729880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:03.191 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:03.191+0000 7f8e7e729880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:03.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:03.207 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:03Z" 2023-09-29T22:11:03.208 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:03.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:08.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:08.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:08.211 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:08.264 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:08.264 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:08.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:08.294+0000 7fb81e132880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:08.295 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:08.295+0000 7fb81e132880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:08.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:08.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:08.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:08.311 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:08Z" 2023-09-29T22:11:08.312 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:08.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:13.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:13.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:13.314 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:13.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:13.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:13.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:13.400+0000 7ff624415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:13.401 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:13.401+0000 7ff624415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:13.412 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:13Z" 2023-09-29T22:11:13.418 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:13.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:18.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:18.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:18.637 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:18.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:18.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:18.724 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:18.724+0000 7f31bfb31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:18.725 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:18.725+0000 7f31bfb31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:18.737 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:18Z" 2023-09-29T22:11:18.743 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:18.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:23.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:23.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:23.746 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:23.799 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:23.799 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:23.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:23.829+0000 7ff41f80a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:23.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:23.830+0000 7ff41f80a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:23.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:23.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:23.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:23.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:23.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:23.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:23.848 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:23.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:23.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:23.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:23.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:23.850 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:23.850 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:23.851 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:23.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:23.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:23.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:23.851 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:23.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:23.853 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:23.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:23.853 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:23Z" 2023-09-29T22:11:23.853 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:23.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:28.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:28.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:28.857 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:28.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:28.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:28.943 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:28.943+0000 7f9314121880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:28.953 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:28.944+0000 7f9314121880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:28.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:28.958 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:28.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:28.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:28.961 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:28.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:28Z" 2023-09-29T22:11:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:28.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:33.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:34.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:34.264 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:34.317 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:34.318 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:34.348 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:34.348+0000 7f495209d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:34.349 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:34.349+0000 7f495209d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:34.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:34Z" 2023-09-29T22:11:34.367 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:34.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1696025381.280108_31.1 target= 2023-09-29T22:11:39.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.370 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:39.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:39.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:39.454 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:39.454+0000 7f7786329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.455 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:39.455+0000 7f7786329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.467 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:39.467 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:39.467 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:39.467 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:39.467 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:39.468 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.470 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.471 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.472 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:39Z" 2023-09-29T22:11:39.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:39.474 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:39.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:39.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.474 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:39.528 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:39.528 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:39.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:39.561+0000 7f4e78134880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:39.562+0000 7f4e78134880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.573 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:39.573 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:39.574 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:39.575 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:39.576 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.577 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:39.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:39Z" 2023-09-29T22:11:39.579 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:39.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:39.580 INFO:rgw_multi.tests:meta checkpoint 2023-09-29T22:11:39.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.580 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:39.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:39.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:39.659 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:39.670 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:39.670 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.670 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:39.670 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.671 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:39.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.672 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:39.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:39.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.673 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:39.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:39.723 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:39.754 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:39.754+0000 7f121aa0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.754 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:39.754+0000 7f121aa0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.767 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.768 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:39.769 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.770 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:39.773 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:39.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:39Z" 2023-09-29T22:11:39.775 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:39.776 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:39.776 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:39.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:39.777 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:39.824 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:39.824 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:39.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:39.853+0000 7f9db994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:39.854+0000 7f9db994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:39.864 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:39.864 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.865 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:39.866 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.867 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:39.869 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:39Z" 2023-09-29T22:11:39.870 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:39.871 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:39.872 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-09-29T22:11:39.890 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.890 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.891 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.891 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.891 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.891 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.892 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.892 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.892 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.893 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.894 DEBUG:boto:Using access key provided by client. 2023-09-29T22:11:39.894 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:11:39.895 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-1 2023-09-29T22:11:39.895 DEBUG:boto:path=/dmuhsu-1/ 2023-09-29T22:11:39.895 DEBUG:boto:auth_path=/dmuhsu-1/ 2023-09-29T22:11:39.895 DEBUG:boto:Method: PUT 2023-09-29T22:11:39.895 DEBUG:boto:Path: /dmuhsu-1/ 2023-09-29T22:11:39.895 DEBUG:boto:Data: 2023-09-29T22:11:39.895 DEBUG:boto:Headers: {} 2023-09-29T22:11:39.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:39.896 DEBUG:boto:Port: 8000 2023-09-29T22:11:39.896 DEBUG:boto:Params: {} 2023-09-29T22:11:39.896 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:11:39.896 DEBUG:boto:Token: None 2023-09-29T22:11:39.896 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:39 GMT /dmuhsu-1/ 2023-09-29T22:11:39.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cxXCMVhUJHShm7yc4rgEWkZi0so= 2023-09-29T22:11:39.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cxXCMVhUJHShm7yc4rgEWkZi0so=', 'Content-Length': '0'} 2023-09-29T22:11:41.982 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000319d450551c87a47-0065174b9b-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:41.983 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-2 2023-09-29T22:11:41.983 DEBUG:boto:path=/dmuhsu-2/ 2023-09-29T22:11:41.983 DEBUG:boto:auth_path=/dmuhsu-2/ 2023-09-29T22:11:41.983 DEBUG:boto:Method: PUT 2023-09-29T22:11:41.983 DEBUG:boto:Path: /dmuhsu-2/ 2023-09-29T22:11:41.983 DEBUG:boto:Data: 2023-09-29T22:11:41.983 DEBUG:boto:Headers: {} 2023-09-29T22:11:41.984 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:41.984 DEBUG:boto:Port: 8000 2023-09-29T22:11:41.984 DEBUG:boto:Params: {} 2023-09-29T22:11:41.984 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:11:41.984 DEBUG:boto:Token: None 2023-09-29T22:11:41.984 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:41 GMT /dmuhsu-2/ 2023-09-29T22:11:41.984 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sdiW0As0gs2oR6biqykmW25BsHA= 2023-09-29T22:11:41.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sdiW0As0gs2oR6biqykmW25BsHA=', 'Content-Length': '0'} 2023-09-29T22:11:44.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0b7c08e7caa0bae-0065174b9d-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:44.965 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-3 2023-09-29T22:11:44.965 DEBUG:boto:path=/dmuhsu-3/ 2023-09-29T22:11:44.965 DEBUG:boto:auth_path=/dmuhsu-3/ 2023-09-29T22:11:44.965 DEBUG:boto:Method: PUT 2023-09-29T22:11:44.965 DEBUG:boto:Path: /dmuhsu-3/ 2023-09-29T22:11:44.965 DEBUG:boto:Data: 2023-09-29T22:11:44.965 DEBUG:boto:Headers: {} 2023-09-29T22:11:44.965 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:44.966 DEBUG:boto:Port: 8001 2023-09-29T22:11:44.966 DEBUG:boto:Params: {} 2023-09-29T22:11:44.966 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:11:44.966 DEBUG:boto:Token: None 2023-09-29T22:11:44.966 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:44 GMT /dmuhsu-3/ 2023-09-29T22:11:44.966 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BbbP4HVcvbVEyWFXr+MxPMfCTDY= 2023-09-29T22:11:44.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BbbP4HVcvbVEyWFXr+MxPMfCTDY=', 'Content-Length': '0'} 2023-09-29T22:11:45.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c252e62d97c32977-0065174ba0-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:45.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:45.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:45.021 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:45.068 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:45.069 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:45.108 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:45.120 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025505.008516_60.1", 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:45.008516Z" 2023-09-29T22:11:45.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:45.122 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:45.122 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:45.123 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:45.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:45.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:45.123 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:45.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:45.176 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:45.206 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:45.206+0000 7faca4924880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:45.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:45.207+0000 7faca4924880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:45.219 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025501.959209_49.1", 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:41.959209Z", 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:45Z" 2023-09-29T22:11:45.226 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:45.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025501.959209_49.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:41.959209Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:45Z" } 2023-09-29T22:11:45.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025501.959209_49.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:41.959209Z', 'realm_epoch': 2}}] 2023-09-29T22:11:45.227 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:45.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025501.959209_49.1'} 2023-09-29T22:11:45.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1696025505.008516_60.1 target=1_1696025501.959209_49.1 2023-09-29T22:11:50.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.229 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:50.282 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:50.283 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:50.316 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:50.317+0000 7f05b5921880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.317 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:50.318+0000 7f05b5921880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.329 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025505.008516_60.1", 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:45.008516Z", 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:50Z" 2023-09-29T22:11:50.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:50.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025505.008516_60.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:45.008516Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:50Z" } 2023-09-29T22:11:50.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025505.008516_60.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:45.008516Z', 'realm_epoch': 2}}] 2023-09-29T22:11:50.337 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:50.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:50.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:50.337 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:50.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.338 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:50.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:50.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:50.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:50.419+0000 7f9dabc13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:50.420+0000 7f9dabc13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.432 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.433 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:50.434 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:50.435 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:50.436 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:50.437 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025505.008516_60.1", 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:45.008516Z", 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:50.438 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:50.439 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:50.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.439 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:50Z" 2023-09-29T22:11:50.439 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:50.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025505.008516_60.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:45.008516Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:50Z" } 2023-09-29T22:11:50.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025505.008516_60.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:45.008516Z', 'realm_epoch': 2}}] 2023-09-29T22:11:50.440 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:50.440 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025505.008516_60.1'} 2023-09-29T22:11:50.440 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:50.441 DEBUG:boto:path=/dmuhsu-1/ 2023-09-29T22:11:50.441 DEBUG:boto:auth_path=/dmuhsu-1/ 2023-09-29T22:11:50.441 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.441 DEBUG:boto:Path: /dmuhsu-1/ 2023-09-29T22:11:50.441 DEBUG:boto:Data: 2023-09-29T22:11:50.441 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.442 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:50.442 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.442 DEBUG:boto:Params: {} 2023-09-29T22:11:50.442 DEBUG:boto:Token: None 2023-09-29T22:11:50.442 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-1/ 2023-09-29T22:11:50.442 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I= 2023-09-29T22:11:50.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I=', 'Content-Length': '0'} 2023-09-29T22:11:50.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e397dbe97f71c995-0065174ba6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.445 DEBUG:boto:path=/dmuhsu-2/ 2023-09-29T22:11:50.445 DEBUG:boto:auth_path=/dmuhsu-2/ 2023-09-29T22:11:50.445 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.445 DEBUG:boto:Path: /dmuhsu-2/ 2023-09-29T22:11:50.445 DEBUG:boto:Data: 2023-09-29T22:11:50.445 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.445 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:50.446 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.446 DEBUG:boto:Params: {} 2023-09-29T22:11:50.446 DEBUG:boto:Token: None 2023-09-29T22:11:50.446 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-2/ 2023-09-29T22:11:50.446 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo= 2023-09-29T22:11:50.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo=', 'Content-Length': '0'} 2023-09-29T22:11:50.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006752180ce00b8ea-0065174ba6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.448 DEBUG:boto:path=/dmuhsu-3/ 2023-09-29T22:11:50.448 DEBUG:boto:auth_path=/dmuhsu-3/ 2023-09-29T22:11:50.448 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.449 DEBUG:boto:Path: /dmuhsu-3/ 2023-09-29T22:11:50.449 DEBUG:boto:Data: 2023-09-29T22:11:50.449 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.449 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:50.449 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.449 DEBUG:boto:Params: {} 2023-09-29T22:11:50.449 DEBUG:boto:Token: None 2023-09-29T22:11:50.449 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-3/ 2023-09-29T22:11:50.450 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8= 2023-09-29T22:11:50.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8=', 'Content-Length': '0'} 2023-09-29T22:11:50.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae95df54289d4a8c-0065174ba6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.452 DEBUG:boto:path=/dmuhsu-1/ 2023-09-29T22:11:50.452 DEBUG:boto:auth_path=/dmuhsu-1/ 2023-09-29T22:11:50.452 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.452 DEBUG:boto:Path: /dmuhsu-1/ 2023-09-29T22:11:50.452 DEBUG:boto:Data: 2023-09-29T22:11:50.452 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.452 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:50.452 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.452 DEBUG:boto:Params: {} 2023-09-29T22:11:50.453 DEBUG:boto:Token: None 2023-09-29T22:11:50.453 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-1/ 2023-09-29T22:11:50.453 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I= 2023-09-29T22:11:50.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I=', 'Content-Length': '0'} 2023-09-29T22:11:50.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b27d3c58ae177f5a-0065174ba6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.455 DEBUG:boto:path=/dmuhsu-2/ 2023-09-29T22:11:50.455 DEBUG:boto:auth_path=/dmuhsu-2/ 2023-09-29T22:11:50.455 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.456 DEBUG:boto:Path: /dmuhsu-2/ 2023-09-29T22:11:50.456 DEBUG:boto:Data: 2023-09-29T22:11:50.456 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.456 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:50.456 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.456 DEBUG:boto:Params: {} 2023-09-29T22:11:50.456 DEBUG:boto:Token: None 2023-09-29T22:11:50.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-2/ 2023-09-29T22:11:50.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo= 2023-09-29T22:11:50.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo=', 'Content-Length': '0'} 2023-09-29T22:11:50.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a029e522bc5f94f2-0065174ba6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.459 DEBUG:boto:path=/dmuhsu-3/ 2023-09-29T22:11:50.459 DEBUG:boto:auth_path=/dmuhsu-3/ 2023-09-29T22:11:50.459 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.459 DEBUG:boto:Path: /dmuhsu-3/ 2023-09-29T22:11:50.459 DEBUG:boto:Data: 2023-09-29T22:11:50.459 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.459 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:50.459 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.459 DEBUG:boto:Params: {} 2023-09-29T22:11:50.460 DEBUG:boto:Token: None 2023-09-29T22:11:50.460 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-3/ 2023-09-29T22:11:50.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8= 2023-09-29T22:11:50.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8=', 'Content-Length': '0'} 2023-09-29T22:11:50.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f79c37c483b57c6b-0065174ba6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.462 DEBUG:boto:path=/dmuhsu-1/ 2023-09-29T22:11:50.462 DEBUG:boto:auth_path=/dmuhsu-1/ 2023-09-29T22:11:50.462 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.462 DEBUG:boto:Path: /dmuhsu-1/ 2023-09-29T22:11:50.462 DEBUG:boto:Data: 2023-09-29T22:11:50.463 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.463 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:50.463 DEBUG:boto:Port: 8001 2023-09-29T22:11:50.463 DEBUG:boto:Params: {} 2023-09-29T22:11:50.463 DEBUG:boto:Token: None 2023-09-29T22:11:50.463 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-1/ 2023-09-29T22:11:50.463 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I= 2023-09-29T22:11:50.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jx2/DZYwizTMReuD8M+RvUnQB7I=', 'Content-Length': '0'} 2023-09-29T22:11:50.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fa22321fcd8df83-0065174ba6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.466 DEBUG:boto:path=/dmuhsu-2/ 2023-09-29T22:11:50.466 DEBUG:boto:auth_path=/dmuhsu-2/ 2023-09-29T22:11:50.466 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.466 DEBUG:boto:Path: /dmuhsu-2/ 2023-09-29T22:11:50.466 DEBUG:boto:Data: 2023-09-29T22:11:50.466 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.466 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:50.466 DEBUG:boto:Port: 8001 2023-09-29T22:11:50.466 DEBUG:boto:Params: {} 2023-09-29T22:11:50.467 DEBUG:boto:Token: None 2023-09-29T22:11:50.467 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-2/ 2023-09-29T22:11:50.467 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo= 2023-09-29T22:11:50.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L69PCPI3NyJ+zSrF1+6FpRDrvxo=', 'Content-Length': '0'} 2023-09-29T22:11:50.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db54fc08638ca56c-0065174ba6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.469 DEBUG:boto:path=/dmuhsu-3/ 2023-09-29T22:11:50.469 DEBUG:boto:auth_path=/dmuhsu-3/ 2023-09-29T22:11:50.469 DEBUG:boto:Method: HEAD 2023-09-29T22:11:50.469 DEBUG:boto:Path: /dmuhsu-3/ 2023-09-29T22:11:50.470 DEBUG:boto:Data: 2023-09-29T22:11:50.470 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.470 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:50.470 DEBUG:boto:Port: 8001 2023-09-29T22:11:50.470 DEBUG:boto:Params: {} 2023-09-29T22:11:50.470 DEBUG:boto:Token: None 2023-09-29T22:11:50.470 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-3/ 2023-09-29T22:11:50.470 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8= 2023-09-29T22:11:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f4cfzX3XekKusdvYdhIaGRWVye8=', 'Content-Length': '0'} 2023-09-29T22:11:50.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d7f06c447304b45-0065174ba6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-09-29T22:11:50.473 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-4 2023-09-29T22:11:50.473 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:50.473 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:50.474 DEBUG:boto:Method: PUT 2023-09-29T22:11:50.474 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:50.474 DEBUG:boto:Data: 2023-09-29T22:11:50.474 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.474 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:50.474 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.474 DEBUG:boto:Params: {} 2023-09-29T22:11:50.474 DEBUG:boto:Token: None 2023-09-29T22:11:50.474 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-4/ 2023-09-29T22:11:50.475 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vDzq8P7eZqcPUwWE0raYgh3K8eY= 2023-09-29T22:11:50.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vDzq8P7eZqcPUwWE0raYgh3K8eY=', 'Content-Length': '0'} 2023-09-29T22:11:50.486 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000581962acf4d66df8-0065174ba6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.487 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-5 2023-09-29T22:11:50.487 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:50.487 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:50.487 DEBUG:boto:Method: PUT 2023-09-29T22:11:50.487 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:50.487 DEBUG:boto:Data: 2023-09-29T22:11:50.487 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.488 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:50.488 DEBUG:boto:Port: 8000 2023-09-29T22:11:50.488 DEBUG:boto:Params: {} 2023-09-29T22:11:50.488 DEBUG:boto:Token: None 2023-09-29T22:11:50.488 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-5/ 2023-09-29T22:11:50.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l2aB5p58C5pTx3ND4Dvmo0sf+Jo= 2023-09-29T22:11:50.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l2aB5p58C5pTx3ND4Dvmo0sf+Jo=', 'Content-Length': '0'} 2023-09-29T22:11:50.511 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000669974f1af3513ca-0065174ba6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.512 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-6 2023-09-29T22:11:50.512 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:50.512 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:50.512 DEBUG:boto:Method: PUT 2023-09-29T22:11:50.512 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:50.512 DEBUG:boto:Data: 2023-09-29T22:11:50.512 DEBUG:boto:Headers: {} 2023-09-29T22:11:50.513 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:50.513 DEBUG:boto:Port: 8001 2023-09-29T22:11:50.513 DEBUG:boto:Params: {} 2023-09-29T22:11:50.513 DEBUG:boto:Token: None 2023-09-29T22:11:50.513 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:50 GMT /dmuhsu-6/ 2023-09-29T22:11:50.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NdhMvQFGRe333rnaJNUT73fKH0I= 2023-09-29T22:11:50.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NdhMvQFGRe333rnaJNUT73fKH0I=', 'Content-Length': '0'} 2023-09-29T22:11:50.534 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096b5332b49927334-0065174ba6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:50.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.535 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:50.581 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:50.581 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:50.610 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:50.621 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:50.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.621 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:50.621 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:50.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:50.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:50.623 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:50.526754Z" 2023-09-29T22:11:50.623 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:50.623 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:50.624 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:50.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:50.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:50.624 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:50.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:50.673 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:50.701 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:50.702+0000 7f989f29d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.702 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:50.703+0000 7f989f29d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:50.718 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:50.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:50.720 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.721 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:50.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:50.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025510.497408_65.1", 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:50.497408Z", 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:50.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:50Z" 2023-09-29T22:11:50.725 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:50.726 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025510.497408_65.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:50.497408Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:50Z" } 2023-09-29T22:11:50.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025510.497408_65.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:50.497408Z', 'realm_epoch': 2}}] 2023-09-29T22:11:50.726 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:50.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.497408_65.1'} 2023-09-29T22:11:50.727 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1696025510.526754_72.1 target=1_1696025510.497408_65.1 2023-09-29T22:11:55.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:55.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:55.728 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:55.781 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:55.781 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:55.812 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:55.811+0000 7fc4de615880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:55.812 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:55.813+0000 7fc4de615880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:55.824 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:55.824 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:55.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:55.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:55.827 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:55.828 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:55.829 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:50.526754Z", 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:55.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:55Z" 2023-09-29T22:11:55.831 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:55.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025510.526754_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:50.526754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:55Z" } 2023-09-29T22:11:55.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025510.526754_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:50.526754Z', 'realm_epoch': 2}}] 2023-09-29T22:11:55.833 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:55.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:55.833 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:55.833 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:55.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:55.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:55.834 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:55.886 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:55.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:55.917+0000 7ffaa091f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:55.918 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:55.918+0000 7ffaa091f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:55.930 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:55.930 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:55.931 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:55.932 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:55.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:55.934 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:50.526754Z", 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:55Z" 2023-09-29T22:11:55.937 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:55.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025510.526754_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:50.526754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:55Z" } 2023-09-29T22:11:55.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025510.526754_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:50.526754Z', 'realm_epoch': 2}}] 2023-09-29T22:11:55.938 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:55.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:55.938 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:55.938 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:55.938 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:55.939 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.939 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:55.939 DEBUG:boto:Data: 2023-09-29T22:11:55.939 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.939 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.939 DEBUG:boto:Params: {} 2023-09-29T22:11:55.939 DEBUG:boto:Token: None 2023-09-29T22:11:55.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-4/ 2023-09-29T22:11:55.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8= 2023-09-29T22:11:55.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8=', 'Content-Length': '0'} 2023-09-29T22:11:55.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e708adaf396680b3-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.943 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:55.943 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:55.943 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.943 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:55.943 DEBUG:boto:Data: 2023-09-29T22:11:55.943 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.944 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.944 DEBUG:boto:Params: {} 2023-09-29T22:11:55.944 DEBUG:boto:Token: None 2023-09-29T22:11:55.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-5/ 2023-09-29T22:11:55.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4= 2023-09-29T22:11:55.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4=', 'Content-Length': '0'} 2023-09-29T22:11:55.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061c326969f29d786-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.946 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:55.946 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:55.947 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.947 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:55.947 DEBUG:boto:Data: 2023-09-29T22:11:55.947 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.947 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.947 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.947 DEBUG:boto:Params: {} 2023-09-29T22:11:55.947 DEBUG:boto:Token: None 2023-09-29T22:11:55.947 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-6/ 2023-09-29T22:11:55.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc= 2023-09-29T22:11:55.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc=', 'Content-Length': '0'} 2023-09-29T22:11:55.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5acfb418895ecf9-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.950 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:55.950 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:55.950 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.950 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:55.950 DEBUG:boto:Data: 2023-09-29T22:11:55.950 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.950 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:55.950 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.950 DEBUG:boto:Params: {} 2023-09-29T22:11:55.950 DEBUG:boto:Token: None 2023-09-29T22:11:55.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-4/ 2023-09-29T22:11:55.951 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8= 2023-09-29T22:11:55.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8=', 'Content-Length': '0'} 2023-09-29T22:11:55.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015d9c0543e5d2dfd-0065174bab-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.953 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:55.954 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:55.954 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.954 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:55.954 DEBUG:boto:Data: 2023-09-29T22:11:55.954 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.954 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:55.954 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.954 DEBUG:boto:Params: {} 2023-09-29T22:11:55.954 DEBUG:boto:Token: None 2023-09-29T22:11:55.955 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-5/ 2023-09-29T22:11:55.955 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4= 2023-09-29T22:11:55.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4=', 'Content-Length': '0'} 2023-09-29T22:11:55.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000999fbc0625661680-0065174bab-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.957 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:55.957 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:55.957 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.957 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:55.957 DEBUG:boto:Data: 2023-09-29T22:11:55.957 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.957 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:55.958 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.958 DEBUG:boto:Params: {} 2023-09-29T22:11:55.958 DEBUG:boto:Token: None 2023-09-29T22:11:55.958 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-6/ 2023-09-29T22:11:55.958 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc= 2023-09-29T22:11:55.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc=', 'Content-Length': '0'} 2023-09-29T22:11:55.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8cc052bf1a2612c-0065174bab-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.960 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:55.960 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:55.960 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.961 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:55.961 DEBUG:boto:Data: 2023-09-29T22:11:55.961 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.961 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:55.961 DEBUG:boto:Port: 8001 2023-09-29T22:11:55.961 DEBUG:boto:Params: {} 2023-09-29T22:11:55.961 DEBUG:boto:Token: None 2023-09-29T22:11:55.961 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-4/ 2023-09-29T22:11:55.961 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8= 2023-09-29T22:11:55.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Khy82rw++W69Qgvds7dPFDa0EQ8=', 'Content-Length': '0'} 2023-09-29T22:11:55.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043af3ac23bb98e19-0065174bab-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.964 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:55.964 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:55.964 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.964 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:55.964 DEBUG:boto:Data: 2023-09-29T22:11:55.964 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:55.964 DEBUG:boto:Port: 8001 2023-09-29T22:11:55.965 DEBUG:boto:Params: {} 2023-09-29T22:11:55.965 DEBUG:boto:Token: None 2023-09-29T22:11:55.965 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-5/ 2023-09-29T22:11:55.965 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4= 2023-09-29T22:11:55.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4io0nmhPyXaq7FXDlgP1b0Ksrw4=', 'Content-Length': '0'} 2023-09-29T22:11:55.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bcf41b9b88b9c5a-0065174bab-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.967 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:55.967 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:55.967 DEBUG:boto:Method: HEAD 2023-09-29T22:11:55.967 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:55.967 DEBUG:boto:Data: 2023-09-29T22:11:55.967 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.967 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:55.968 DEBUG:boto:Port: 8001 2023-09-29T22:11:55.968 DEBUG:boto:Params: {} 2023-09-29T22:11:55.968 DEBUG:boto:Token: None 2023-09-29T22:11:55.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-6/ 2023-09-29T22:11:55.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc= 2023-09-29T22:11:55.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+UQW1nlIJs1OiOLI83bx0Jyh0Pc=', 'Content-Length': '0'} 2023-09-29T22:11:55.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a54736e737b0ca7e-0065174bab-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.970 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:55.970 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:55.971 DEBUG:boto:Method: PUT 2023-09-29T22:11:55.971 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:55.971 DEBUG:boto:Data: 2023-09-29T22:11:55.971 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.971 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.971 DEBUG:boto:Params: {} 2023-09-29T22:11:55.971 DEBUG:boto:Token: None 2023-09-29T22:11:55.971 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-4/ 2023-09-29T22:11:55.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AgLJLlExtRHTuPlK4HLbx1n0ZxU= 2023-09-29T22:11:55.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AgLJLlExtRHTuPlK4HLbx1n0ZxU=', 'Content-Length': '0'} 2023-09-29T22:11:55.974 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000797dc708fb36e876-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.974 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:55.974 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:55.974 DEBUG:boto:Method: PUT 2023-09-29T22:11:55.975 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:55.975 DEBUG:boto:Data: 2023-09-29T22:11:55.975 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.975 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.975 DEBUG:boto:Params: {} 2023-09-29T22:11:55.975 DEBUG:boto:Token: None 2023-09-29T22:11:55.975 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-5/ 2023-09-29T22:11:55.975 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZUBGv/q/MMOlXUi4XZ2cWYbfccM= 2023-09-29T22:11:55.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZUBGv/q/MMOlXUi4XZ2cWYbfccM=', 'Content-Length': '0'} 2023-09-29T22:11:55.978 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5ebc8759dbec9d8-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.978 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:55.978 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:55.978 DEBUG:boto:Method: PUT 2023-09-29T22:11:55.978 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:55.978 DEBUG:boto:Data: 2023-09-29T22:11:55.978 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.979 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:55.979 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.979 DEBUG:boto:Params: {} 2023-09-29T22:11:55.979 DEBUG:boto:Token: None 2023-09-29T22:11:55.979 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-6/ 2023-09-29T22:11:55.979 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nuXtXY/6nlWPaBBTQCteTh/lAGw= 2023-09-29T22:11:55.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nuXtXY/6nlWPaBBTQCteTh/lAGw=', 'Content-Length': '0'} 2023-09-29T22:11:55.982 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011c89adc4b3bed21-0065174bab-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.982 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:55.982 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:55.982 DEBUG:boto:Method: PUT 2023-09-29T22:11:55.982 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:55.982 DEBUG:boto:Data: 2023-09-29T22:11:55.982 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.982 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:55.982 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.983 DEBUG:boto:Params: {} 2023-09-29T22:11:55.983 DEBUG:boto:Token: None 2023-09-29T22:11:55.983 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-4/ 2023-09-29T22:11:55.983 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AgLJLlExtRHTuPlK4HLbx1n0ZxU= 2023-09-29T22:11:55.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AgLJLlExtRHTuPlK4HLbx1n0ZxU=', 'Content-Length': '0'} 2023-09-29T22:11:55.988 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000216d0aa7f5f9707f-0065174bab-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:55.988 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:55.988 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:55.989 DEBUG:boto:Method: PUT 2023-09-29T22:11:55.989 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:55.989 DEBUG:boto:Data: 2023-09-29T22:11:55.989 DEBUG:boto:Headers: {} 2023-09-29T22:11:55.989 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:55.989 DEBUG:boto:Port: 8000 2023-09-29T22:11:55.989 DEBUG:boto:Params: {} 2023-09-29T22:11:55.989 DEBUG:boto:Token: None 2023-09-29T22:11:55.989 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:55 GMT /dmuhsu-5/ 2023-09-29T22:11:55.989 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZUBGv/q/MMOlXUi4XZ2cWYbfccM= 2023-09-29T22:11:55.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZUBGv/q/MMOlXUi4XZ2cWYbfccM=', 'Content-Length': '0'} 2023-09-29T22:11:56.035 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf44472cda2bff19-0065174bab-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.035 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.035 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.036 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.036 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.036 DEBUG:boto:Data: 2023-09-29T22:11:56.036 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.036 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.036 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.036 DEBUG:boto:Params: {} 2023-09-29T22:11:56.036 DEBUG:boto:Token: None 2023-09-29T22:11:56.036 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.037 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FZu+XpyLkAhCnr+PNQoVBdzF4hU= 2023-09-29T22:11:56.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FZu+XpyLkAhCnr+PNQoVBdzF4hU=', 'Content-Length': '0'} 2023-09-29T22:11:56.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a11c9331b1979341-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.082 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.083 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.083 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.083 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.083 DEBUG:boto:Data: 2023-09-29T22:11:56.083 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.083 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.083 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.083 DEBUG:boto:Params: {} 2023-09-29T22:11:56.083 DEBUG:boto:Token: None 2023-09-29T22:11:56.084 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tMjE+dnSHFYTR5q7paGP9jT9ec= 2023-09-29T22:11:56.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tMjE+dnSHFYTR5q7paGP9jT9ec=', 'Content-Length': '0'} 2023-09-29T22:11:56.088 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bae9e4b5b7a80b86-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.089 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.089 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.089 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.089 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.089 DEBUG:boto:Data: 2023-09-29T22:11:56.089 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.089 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.089 DEBUG:boto:Params: {} 2023-09-29T22:11:56.090 DEBUG:boto:Token: None 2023-09-29T22:11:56.090 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aoG0uPO1sNu7ruRUGdCiwxy6pd0= 2023-09-29T22:11:56.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aoG0uPO1sNu7ruRUGdCiwxy6pd0=', 'Content-Length': '0'} 2023-09-29T22:11:56.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000049108183110cca68-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.136 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.136 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.136 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.136 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.136 DEBUG:boto:Data: 2023-09-29T22:11:56.136 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.136 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.136 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.137 DEBUG:boto:Params: {} 2023-09-29T22:11:56.137 DEBUG:boto:Token: None 2023-09-29T22:11:56.137 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.137 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FZu+XpyLkAhCnr+PNQoVBdzF4hU= 2023-09-29T22:11:56.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FZu+XpyLkAhCnr+PNQoVBdzF4hU=', 'Content-Length': '0'} 2023-09-29T22:11:56.182 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d2f8c36447e85171-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.183 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.183 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.183 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.183 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.183 DEBUG:boto:Data: 2023-09-29T22:11:56.183 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.183 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.183 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.183 DEBUG:boto:Params: {} 2023-09-29T22:11:56.184 DEBUG:boto:Token: None 2023-09-29T22:11:56.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd8d91025c23958e-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.186 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.186 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.186 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.186 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.186 DEBUG:boto:Data: 2023-09-29T22:11:56.187 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.187 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.187 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.187 DEBUG:boto:Params: {} 2023-09-29T22:11:56.187 DEBUG:boto:Token: None 2023-09-29T22:11:56.187 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.187 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1bdcf4781f34b44-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.189 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.189 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.189 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.189 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.190 DEBUG:boto:Data: 2023-09-29T22:11:56.190 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.190 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.190 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.190 DEBUG:boto:Params: {} 2023-09-29T22:11:56.190 DEBUG:boto:Token: None 2023-09-29T22:11:56.190 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.190 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfbd2abf39912bb8-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.192 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.192 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.193 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.193 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.193 DEBUG:boto:Data: 2023-09-29T22:11:56.193 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.193 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.193 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.193 DEBUG:boto:Params: {} 2023-09-29T22:11:56.193 DEBUG:boto:Token: None 2023-09-29T22:11:56.193 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.194 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095afcfe9bc393cb9-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.196 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.196 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.196 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.197 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.197 DEBUG:boto:Data: 2023-09-29T22:11:56.197 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.197 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.197 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.197 DEBUG:boto:Params: {} 2023-09-29T22:11:56.197 DEBUG:boto:Token: None 2023-09-29T22:11:56.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebab7c6ce0418261-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.200 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.200 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.200 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.200 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.200 DEBUG:boto:Data: 2023-09-29T22:11:56.200 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.200 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.200 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.200 DEBUG:boto:Params: {} 2023-09-29T22:11:56.200 DEBUG:boto:Token: None 2023-09-29T22:11:56.201 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d56a5b631da6520-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.203 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.203 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.203 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.203 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.203 DEBUG:boto:Data: 2023-09-29T22:11:56.203 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.204 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.204 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.204 DEBUG:boto:Params: {} 2023-09-29T22:11:56.204 DEBUG:boto:Token: None 2023-09-29T22:11:56.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d4bba0d581a68c1-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.206 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.206 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.207 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.207 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.207 DEBUG:boto:Data: 2023-09-29T22:11:56.207 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.207 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.207 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.207 DEBUG:boto:Params: {} 2023-09-29T22:11:56.207 DEBUG:boto:Token: None 2023-09-29T22:11:56.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d60c555e2f45c3b-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.209 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.210 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.210 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.210 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.210 DEBUG:boto:Data: 2023-09-29T22:11:56.210 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.210 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.210 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.210 DEBUG:boto:Params: {} 2023-09-29T22:11:56.210 DEBUG:boto:Token: None 2023-09-29T22:11:56.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.211 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017b164a69fc928e-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.213 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.257 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.286 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.297 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:50.526754Z" 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.298 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:56.299 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:56.299 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:56.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.300 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.352 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.383 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:56.382+0000 7fe98972a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.383 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:56.383+0000 7fe98972a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:56.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:56.395 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:50.526754Z", 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:56.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.402 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:56Z" 2023-09-29T22:11:56.402 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:56.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025510.526754_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:50.526754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:56Z" } 2023-09-29T22:11:56.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025510.526754_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:50.526754Z', 'realm_epoch': 2}}] 2023-09-29T22:11:56.403 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:56.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:56.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:56.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:56.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.404 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.451 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.452 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:56.477+0000 7f2ee2d20880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:56.477+0000 7f2ee2d20880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.489 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:56.489 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:56.489 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:56.489 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:56.489 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.490 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:56.491 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025510.526754_72.1", 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.494 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:50.526754Z", 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.495 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:56Z" 2023-09-29T22:11:56.496 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:56.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025510.526754_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:50.526754Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:56Z" } 2023-09-29T22:11:56.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025510.526754_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:50.526754Z', 'realm_epoch': 2}}] 2023-09-29T22:11:56.497 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:56.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025510.526754_72.1'} 2023-09-29T22:11:56.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:56.497 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.497 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.498 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.498 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.498 DEBUG:boto:Data: 2023-09-29T22:11:56.498 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.498 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.498 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.498 DEBUG:boto:Params: {} 2023-09-29T22:11:56.498 DEBUG:boto:Token: None 2023-09-29T22:11:56.499 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.499 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009be1136d3c031196-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.501 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.501 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.501 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.501 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.502 DEBUG:boto:Data: 2023-09-29T22:11:56.502 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.502 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.502 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.502 DEBUG:boto:Params: {} 2023-09-29T22:11:56.502 DEBUG:boto:Token: None 2023-09-29T22:11:56.502 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.502 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c35190d0b2b1e69a-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.504 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.504 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.505 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.505 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.505 DEBUG:boto:Data: 2023-09-29T22:11:56.505 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.505 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.505 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.505 DEBUG:boto:Params: {} 2023-09-29T22:11:56.505 DEBUG:boto:Token: None 2023-09-29T22:11:56.505 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ce56bcbc07bf2da-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.507 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.508 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.508 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.508 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.508 DEBUG:boto:Data: 2023-09-29T22:11:56.508 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.508 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.508 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.508 DEBUG:boto:Params: {} 2023-09-29T22:11:56.508 DEBUG:boto:Token: None 2023-09-29T22:11:56.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037db1dcec598476d-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.512 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.512 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.512 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.512 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.512 DEBUG:boto:Data: 2023-09-29T22:11:56.512 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.512 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.512 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.512 DEBUG:boto:Params: {} 2023-09-29T22:11:56.512 DEBUG:boto:Token: None 2023-09-29T22:11:56.513 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdac5e89a83713b3-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.515 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.515 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.515 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.515 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.515 DEBUG:boto:Data: 2023-09-29T22:11:56.515 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.516 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.516 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.516 DEBUG:boto:Params: {} 2023-09-29T22:11:56.516 DEBUG:boto:Token: None 2023-09-29T22:11:56.516 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068a7a2b408b8d79f-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.518 DEBUG:boto:path=/dmuhsu-4/ 2023-09-29T22:11:56.518 DEBUG:boto:auth_path=/dmuhsu-4/ 2023-09-29T22:11:56.519 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.519 DEBUG:boto:Path: /dmuhsu-4/ 2023-09-29T22:11:56.519 DEBUG:boto:Data: 2023-09-29T22:11:56.519 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.519 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.519 DEBUG:boto:Params: {} 2023-09-29T22:11:56.519 DEBUG:boto:Token: None 2023-09-29T22:11:56.519 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-4/ 2023-09-29T22:11:56.519 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ= 2023-09-29T22:11:56.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:exVTLRgFaG+Vyo6gPvlr0h9e7FQ=', 'Content-Length': '0'} 2023-09-29T22:11:56.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec41bc3a6e561ce2-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.522 DEBUG:boto:path=/dmuhsu-5/ 2023-09-29T22:11:56.522 DEBUG:boto:auth_path=/dmuhsu-5/ 2023-09-29T22:11:56.522 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.522 DEBUG:boto:Path: /dmuhsu-5/ 2023-09-29T22:11:56.522 DEBUG:boto:Data: 2023-09-29T22:11:56.522 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.522 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.522 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.522 DEBUG:boto:Params: {} 2023-09-29T22:11:56.523 DEBUG:boto:Token: None 2023-09-29T22:11:56.523 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-5/ 2023-09-29T22:11:56.523 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw= 2023-09-29T22:11:56.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QYRZ/IkjzirMcQqMO7HSLiFwhfw=', 'Content-Length': '0'} 2023-09-29T22:11:56.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008226c32175109b94-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.525 DEBUG:boto:path=/dmuhsu-6/ 2023-09-29T22:11:56.525 DEBUG:boto:auth_path=/dmuhsu-6/ 2023-09-29T22:11:56.525 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.525 DEBUG:boto:Path: /dmuhsu-6/ 2023-09-29T22:11:56.525 DEBUG:boto:Data: 2023-09-29T22:11:56.525 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.525 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.526 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.526 DEBUG:boto:Params: {} 2023-09-29T22:11:56.526 DEBUG:boto:Token: None 2023-09-29T22:11:56.526 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-6/ 2023-09-29T22:11:56.526 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o= 2023-09-29T22:11:56.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtfhDGbY1Q3yaW0mRiPZcru447o=', 'Content-Length': '0'} 2023-09-29T22:11:56.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e522c5fb0780fa1-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-09-29T22:11:56.529 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-7 2023-09-29T22:11:56.529 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:11:56.529 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:11:56.529 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.529 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:11:56.529 DEBUG:boto:Data: 2023-09-29T22:11:56.529 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.529 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.530 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.530 DEBUG:boto:Params: {} 2023-09-29T22:11:56.530 DEBUG:boto:Token: None 2023-09-29T22:11:56.530 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-7/ 2023-09-29T22:11:56.530 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JyV9k3q+RYVE/bSlG//AlLm2Tdk= 2023-09-29T22:11:56.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JyV9k3q+RYVE/bSlG//AlLm2Tdk=', 'Content-Length': '0'} 2023-09-29T22:11:56.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b2544962ae09d90-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.549 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-8 2023-09-29T22:11:56.549 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:11:56.550 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:11:56.550 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.550 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:11:56.550 DEBUG:boto:Data: 2023-09-29T22:11:56.550 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.550 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.550 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.550 DEBUG:boto:Params: {} 2023-09-29T22:11:56.551 DEBUG:boto:Token: None 2023-09-29T22:11:56.551 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-8/ 2023-09-29T22:11:56.551 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y4pWKBSNngOcq0JFkwugcNVO9NE= 2023-09-29T22:11:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y4pWKBSNngOcq0JFkwugcNVO9NE=', 'Content-Length': '0'} 2023-09-29T22:11:56.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a311cfa6c06aee15-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.579 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-9 2023-09-29T22:11:56.579 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:11:56.579 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:11:56.579 DEBUG:boto:Method: PUT 2023-09-29T22:11:56.580 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:11:56.580 DEBUG:boto:Data: 2023-09-29T22:11:56.580 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.580 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.580 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.580 DEBUG:boto:Params: {} 2023-09-29T22:11:56.580 DEBUG:boto:Token: None 2023-09-29T22:11:56.580 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-9/ 2023-09-29T22:11:56.580 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oDzWx6BS98VX3jKT6jWWN68oW/A= 2023-09-29T22:11:56.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oDzWx6BS98VX3jKT6jWWN68oW/A=', 'Content-Length': '0'} 2023-09-29T22:11:56.604 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000362c8b7af440fd6b-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.605 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.649 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.649 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.679 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:56.690 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:56.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.690 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.690 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:56.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:56.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025516.597203_85.1", 2023-09-29T22:11:56.692 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:56.597203Z" 2023-09-29T22:11:56.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.692 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:56.693 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:56.693 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:56.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.694 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.783 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:56.783+0000 7f12875a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.784 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:56.784+0000 7f12875a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.796 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:56.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.798 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:56.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025516.597203_85.1", 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:56.597203Z", 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:56.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:56.802 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:56.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:56.802 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:56Z" 2023-09-29T22:11:56.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:56.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025516.597203_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:56.597203Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:56Z" } 2023-09-29T22:11:56.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025516.597203_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:56.597203Z', 'realm_epoch': 2}}] 2023-09-29T22:11:56.803 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:56.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:56.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:11:56.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:11:56.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:56.804 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:56.849 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:56.879 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:56.878+0000 7fe08fb26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.880 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:11:56.879+0000 7fe08fb26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.893 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.894 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.895 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:56.896 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:11:56.897 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025516.597203_85.1", 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:56.597203Z", 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.898 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:11:56Z" 2023-09-29T22:11:56.899 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:11:56.900 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025516.597203_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:56.597203Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:56Z" } 2023-09-29T22:11:56.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025516.597203_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:56.597203Z', 'realm_epoch': 2}}] 2023-09-29T22:11:56.901 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:56.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:56.901 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:11:56.901 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:11:56.901 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:11:56.902 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.902 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:11:56.902 DEBUG:boto:Data: 2023-09-29T22:11:56.902 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.902 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.902 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.902 DEBUG:boto:Params: {} 2023-09-29T22:11:56.902 DEBUG:boto:Token: None 2023-09-29T22:11:56.902 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-7/ 2023-09-29T22:11:56.903 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc= 2023-09-29T22:11:56.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc=', 'Content-Length': '0'} 2023-09-29T22:11:56.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000942dac303452fb0e-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.905 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:11:56.905 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:11:56.905 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.906 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:11:56.906 DEBUG:boto:Data: 2023-09-29T22:11:56.906 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.906 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.906 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.906 DEBUG:boto:Params: {} 2023-09-29T22:11:56.906 DEBUG:boto:Token: None 2023-09-29T22:11:56.906 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-8/ 2023-09-29T22:11:56.906 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY= 2023-09-29T22:11:56.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY=', 'Content-Length': '0'} 2023-09-29T22:11:56.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052d14c3d8a84ddd2-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.909 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:11:56.909 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:11:56.909 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.909 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:11:56.909 DEBUG:boto:Data: 2023-09-29T22:11:56.909 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.909 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.910 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.910 DEBUG:boto:Params: {} 2023-09-29T22:11:56.910 DEBUG:boto:Token: None 2023-09-29T22:11:56.910 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-9/ 2023-09-29T22:11:56.910 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE= 2023-09-29T22:11:56.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE=', 'Content-Length': '0'} 2023-09-29T22:11:56.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009da7e8907e8d9886-0065174bac-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.912 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:11:56.912 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:11:56.912 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.913 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:11:56.913 DEBUG:boto:Data: 2023-09-29T22:11:56.913 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.913 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.914 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.914 DEBUG:boto:Params: {} 2023-09-29T22:11:56.914 DEBUG:boto:Token: None 2023-09-29T22:11:56.914 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-7/ 2023-09-29T22:11:56.914 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc= 2023-09-29T22:11:56.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc=', 'Content-Length': '0'} 2023-09-29T22:11:56.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065d1e98c5931d726-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.917 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:11:56.917 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:11:56.917 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.917 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:11:56.917 DEBUG:boto:Data: 2023-09-29T22:11:56.917 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.917 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.917 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.917 DEBUG:boto:Params: {} 2023-09-29T22:11:56.918 DEBUG:boto:Token: None 2023-09-29T22:11:56.918 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-8/ 2023-09-29T22:11:56.918 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY= 2023-09-29T22:11:56.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY=', 'Content-Length': '0'} 2023-09-29T22:11:56.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a921fbf747c88c2-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.920 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:11:56.920 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:11:56.920 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.920 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:11:56.920 DEBUG:boto:Data: 2023-09-29T22:11:56.920 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.921 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.921 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.921 DEBUG:boto:Params: {} 2023-09-29T22:11:56.921 DEBUG:boto:Token: None 2023-09-29T22:11:56.921 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-9/ 2023-09-29T22:11:56.921 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE= 2023-09-29T22:11:56.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE=', 'Content-Length': '0'} 2023-09-29T22:11:56.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fda0a27283b435e-0065174bac-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.923 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:11:56.924 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:11:56.924 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.924 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:11:56.924 DEBUG:boto:Data: 2023-09-29T22:11:56.924 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.924 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.924 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.924 DEBUG:boto:Params: {} 2023-09-29T22:11:56.924 DEBUG:boto:Token: None 2023-09-29T22:11:56.925 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-7/ 2023-09-29T22:11:56.925 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc= 2023-09-29T22:11:56.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:krFO48xIVLZ+KLRM8/daQBAH1zc=', 'Content-Length': '0'} 2023-09-29T22:11:56.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0b6f02f11cc2ca0-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.927 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:11:56.927 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:11:56.927 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.927 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:11:56.928 DEBUG:boto:Data: 2023-09-29T22:11:56.928 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.928 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.928 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.928 DEBUG:boto:Params: {} 2023-09-29T22:11:56.928 DEBUG:boto:Token: None 2023-09-29T22:11:56.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-8/ 2023-09-29T22:11:56.929 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY= 2023-09-29T22:11:56.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tVSkFs0JI6bEtN/pissClD8Z7CY=', 'Content-Length': '0'} 2023-09-29T22:11:56.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc0209276348a9f4-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.931 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:11:56.931 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:11:56.931 DEBUG:boto:Method: HEAD 2023-09-29T22:11:56.931 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:11:56.931 DEBUG:boto:Data: 2023-09-29T22:11:56.931 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.931 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:56.931 DEBUG:boto:Port: 8001 2023-09-29T22:11:56.931 DEBUG:boto:Params: {} 2023-09-29T22:11:56.932 DEBUG:boto:Token: None 2023-09-29T22:11:56.932 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-9/ 2023-09-29T22:11:56.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE= 2023-09-29T22:11:56.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vIl2Isw75eg8ZD+l6l7flUELZeE=', 'Content-Length': '0'} 2023-09-29T22:11:56.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d871bdf9c51b46c5-0065174bac-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.934 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:11:56.934 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:11:56.934 DEBUG:boto:Method: DELETE 2023-09-29T22:11:56.934 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:11:56.934 DEBUG:boto:Data: 2023-09-29T22:11:56.934 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.934 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:11:56.934 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.935 DEBUG:boto:Params: {} 2023-09-29T22:11:56.935 DEBUG:boto:Token: None 2023-09-29T22:11:56.935 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-7/ 2023-09-29T22:11:56.935 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X1kXFrT1NVqZIhptcRplCT52vYE= 2023-09-29T22:11:56.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X1kXFrT1NVqZIhptcRplCT52vYE=', 'Content-Length': '0'} 2023-09-29T22:11:56.959 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083a3d31d62652ff3-0065174bac-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:11:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:56.960 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:11:56.960 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:11:56.960 DEBUG:boto:Method: DELETE 2023-09-29T22:11:56.960 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:11:56.960 DEBUG:boto:Data: 2023-09-29T22:11:56.960 DEBUG:boto:Headers: {} 2023-09-29T22:11:56.960 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:11:56.960 DEBUG:boto:Port: 8000 2023-09-29T22:11:56.960 DEBUG:boto:Params: {} 2023-09-29T22:11:56.961 DEBUG:boto:Token: None 2023-09-29T22:11:56.961 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:11:56 GMT /dmuhsu-8/ 2023-09-29T22:11:56.961 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wvZjG3z619QRYeBBM8uRqCwUqCs= 2023-09-29T22:11:56.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wvZjG3z619QRYeBBM8uRqCwUqCs=', 'Content-Length': '0'} 2023-09-29T22:11:57.016 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5d579cf0d0d8ea9-0065174bac-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:11:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:57.016 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:11:57.016 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:11:57.016 DEBUG:boto:Method: DELETE 2023-09-29T22:11:57.017 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:11:57.017 DEBUG:boto:Data: 2023-09-29T22:11:57.017 DEBUG:boto:Headers: {} 2023-09-29T22:11:57.017 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:11:57.017 DEBUG:boto:Port: 8001 2023-09-29T22:11:57.017 DEBUG:boto:Params: {} 2023-09-29T22:11:57.017 DEBUG:boto:Token: None 2023-09-29T22:11:57.017 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:11:57 GMT /dmuhsu-9/ 2023-09-29T22:11:57.017 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lTgZn4JgKqj08gtZYcBVQcn2aIQ= 2023-09-29T22:11:57.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:11:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lTgZn4JgKqj08gtZYcBVQcn2aIQ=', 'Content-Length': '0'} 2023-09-29T22:11:57.071 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a8f85887ed3cca0-0065174bad-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:11:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:11:57.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:57.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:57.072 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:57.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:57.120 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:57.148 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:57.163 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:11:57.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:11:57.165 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:11:57.165 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:11:57.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:11:57.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:57.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:11:57.166 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:11:57.211 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:57.212 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:57.242 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:57.242+0000 7fddadd37880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:57.242 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:11:57.242+0000 7fddadd37880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:11:57.254 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:11:57.255 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:57.256 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:57.257 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:11:57.258 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025516.597203_85.1", 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:56.597203Z", 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:57.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:11:57Z" 2023-09-29T22:11:57.260 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:11:57.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025516.597203_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:56.597203Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:11:57Z" } 2023-09-29T22:11:57.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025516.597203_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:56.597203Z', 'realm_epoch': 2}}] 2023-09-29T22:11:57.262 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:11:57.262 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.597203_85.1'} 2023-09-29T22:11:57.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1696025517.047167_98.1 target=1_1696025516.597203_85.1 2023-09-29T22:12:02.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:02.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:02.264 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:02.319 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:02.319 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:02.350 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:02.350+0000 7f28c994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:02.351 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:02.351+0000 7f28c994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:02.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:12:02.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:02.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:02.366 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:02.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025516.945967_86.1", 2023-09-29T22:12:02.368 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:56.945967Z", 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:02.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:02.370 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:12:02.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.370 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:12:02.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:12:02.371 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:12:02.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:02.371 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:12:02Z" 2023-09-29T22:12:02.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:12:02.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025516.945967_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:56.945967Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:02Z" } 2023-09-29T22:12:02.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025516.945967_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:56.945967Z', 'realm_epoch': 2}}] 2023-09-29T22:12:02.373 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:02.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025516.945967_86.1'} 2023-09-29T22:12:02.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1696025517.047167_98.1 target=1_1696025516.945967_86.1 2023-09-29T22:12:07.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.375 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:07.427 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:07.428 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:07.459 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:07.459+0000 7f0489935880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:07.459 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:07.460+0000 7f0489935880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:07.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:12:07.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:07.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:07.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:07.475 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:07.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:12:07.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:12:07.478 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:12:07.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:07.478 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:12:07Z" 2023-09-29T22:12:07.478 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:12:07.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:07Z" } 2023-09-29T22:12:07.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:07.479 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:07.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:07.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:12:07.479 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:12:07.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.480 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:07.532 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:07.532 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:07.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:07.564+0000 7f3a54c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:07.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:07.565+0000 7f3a54c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:07.576 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:07.577 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:07.578 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:07.579 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:07.580 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:07.581 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:07.582 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:12:07Z" 2023-09-29T22:12:07.583 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:12:07.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:07Z" } 2023-09-29T22:12:07.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:07.585 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:07.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:07.585 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:12:07.585 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:12:07.585 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:12:07.585 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.585 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:12:07.585 DEBUG:boto:Data: 2023-09-29T22:12:07.585 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.586 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.586 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.586 DEBUG:boto:Params: {} 2023-09-29T22:12:07.586 DEBUG:boto:Token: None 2023-09-29T22:12:07.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-7/ 2023-09-29T22:12:07.586 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g= 2023-09-29T22:12:07.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g=', 'Content-Length': '0'} 2023-09-29T22:12:07.587 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000004539cc47963ef0c5-0065174bb7-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.588 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:12:07.588 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:12:07.588 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.588 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:12:07.588 DEBUG:boto:Data: 2023-09-29T22:12:07.588 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.588 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.588 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.588 DEBUG:boto:Params: {} 2023-09-29T22:12:07.589 DEBUG:boto:Token: None 2023-09-29T22:12:07.589 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-8/ 2023-09-29T22:12:07.589 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM= 2023-09-29T22:12:07.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM=', 'Content-Length': '0'} 2023-09-29T22:12:07.590 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000820ff579433c30ed-0065174bb7-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.590 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:12:07.590 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:12:07.590 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.591 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:12:07.591 DEBUG:boto:Data: 2023-09-29T22:12:07.591 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.591 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.591 DEBUG:boto:Params: {} 2023-09-29T22:12:07.591 DEBUG:boto:Token: None 2023-09-29T22:12:07.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-9/ 2023-09-29T22:12:07.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ= 2023-09-29T22:12:07.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ=', 'Content-Length': '0'} 2023-09-29T22:12:07.592 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000f5a29ac7b638e3e8-0065174bb7-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.593 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:12:07.593 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:12:07.593 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.593 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:12:07.593 DEBUG:boto:Data: 2023-09-29T22:12:07.593 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.593 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.593 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.593 DEBUG:boto:Params: {} 2023-09-29T22:12:07.593 DEBUG:boto:Token: None 2023-09-29T22:12:07.594 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-7/ 2023-09-29T22:12:07.594 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g= 2023-09-29T22:12:07.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g=', 'Content-Length': '0'} 2023-09-29T22:12:07.595 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000031c160965653a556-0065174bb7-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.595 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:12:07.595 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:12:07.595 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.595 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:12:07.596 DEBUG:boto:Data: 2023-09-29T22:12:07.596 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.596 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.596 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.596 DEBUG:boto:Params: {} 2023-09-29T22:12:07.596 DEBUG:boto:Token: None 2023-09-29T22:12:07.596 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-8/ 2023-09-29T22:12:07.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM= 2023-09-29T22:12:07.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM=', 'Content-Length': '0'} 2023-09-29T22:12:07.597 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000095c6daea0d836ad0-0065174bb7-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.598 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:12:07.598 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:12:07.598 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.598 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:12:07.598 DEBUG:boto:Data: 2023-09-29T22:12:07.598 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.598 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.598 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.598 DEBUG:boto:Params: {} 2023-09-29T22:12:07.599 DEBUG:boto:Token: None 2023-09-29T22:12:07.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-9/ 2023-09-29T22:12:07.599 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ= 2023-09-29T22:12:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ=', 'Content-Length': '0'} 2023-09-29T22:12:07.600 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000a0de5cc7c9c39a5d-0065174bb7-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.600 DEBUG:boto:path=/dmuhsu-7/ 2023-09-29T22:12:07.600 DEBUG:boto:auth_path=/dmuhsu-7/ 2023-09-29T22:12:07.601 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.601 DEBUG:boto:Path: /dmuhsu-7/ 2023-09-29T22:12:07.601 DEBUG:boto:Data: 2023-09-29T22:12:07.601 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.601 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.601 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.601 DEBUG:boto:Params: {} 2023-09-29T22:12:07.601 DEBUG:boto:Token: None 2023-09-29T22:12:07.601 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-7/ 2023-09-29T22:12:07.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g= 2023-09-29T22:12:07.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hz6BqTcaRKmJL71lebEX0i80g2g=', 'Content-Length': '0'} 2023-09-29T22:12:07.603 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx000001a068f656146c02f-0065174bb7-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.603 DEBUG:boto:path=/dmuhsu-8/ 2023-09-29T22:12:07.603 DEBUG:boto:auth_path=/dmuhsu-8/ 2023-09-29T22:12:07.603 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.603 DEBUG:boto:Path: /dmuhsu-8/ 2023-09-29T22:12:07.603 DEBUG:boto:Data: 2023-09-29T22:12:07.603 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.604 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.604 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.604 DEBUG:boto:Params: {} 2023-09-29T22:12:07.604 DEBUG:boto:Token: None 2023-09-29T22:12:07.604 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-8/ 2023-09-29T22:12:07.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM= 2023-09-29T22:12:07.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wzwijX1O6AE1dkF2hwdtMXJZUTM=', 'Content-Length': '0'} 2023-09-29T22:12:07.605 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000010fc8e6625dfae8e-0065174bb7-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.605 DEBUG:boto:path=/dmuhsu-9/ 2023-09-29T22:12:07.605 DEBUG:boto:auth_path=/dmuhsu-9/ 2023-09-29T22:12:07.605 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.605 DEBUG:boto:Path: /dmuhsu-9/ 2023-09-29T22:12:07.606 DEBUG:boto:Data: 2023-09-29T22:12:07.606 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.606 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.606 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.606 DEBUG:boto:Params: {} 2023-09-29T22:12:07.606 DEBUG:boto:Token: None 2023-09-29T22:12:07.606 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-9/ 2023-09-29T22:12:07.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ= 2023-09-29T22:12:07.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9E6t+o/0ylOGSG/zl+U57HZlrPQ=', 'Content-Length': '0'} 2023-09-29T22:12:07.607 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000c4f6f01b4c429353-0065174bb7-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-09-29T22:12:07.608 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-10 2023-09-29T22:12:07.608 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:07.609 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:07.609 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.609 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:07.609 DEBUG:boto:Data: 2023-09-29T22:12:07.609 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.609 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.609 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.609 DEBUG:boto:Params: {} 2023-09-29T22:12:07.609 DEBUG:boto:Token: None 2023-09-29T22:12:07.610 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/ 2023-09-29T22:12:07.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GuyRSOtf0pj9SkkbHR5r26djFm0= 2023-09-29T22:12:07.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GuyRSOtf0pj9SkkbHR5r26djFm0=', 'Content-Length': '0'} 2023-09-29T22:12:07.622 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f8f03e28e5613b4-0065174bb7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.623 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-11 2023-09-29T22:12:07.623 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:07.623 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:07.623 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.623 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:07.624 DEBUG:boto:Data: 2023-09-29T22:12:07.624 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.624 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.624 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.624 DEBUG:boto:Params: {} 2023-09-29T22:12:07.625 DEBUG:boto:Token: None 2023-09-29T22:12:07.625 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/ 2023-09-29T22:12:07.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QUgesqV1MFAruAVWs2rix7oUxj8= 2023-09-29T22:12:07.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QUgesqV1MFAruAVWs2rix7oUxj8=', 'Content-Length': '0'} 2023-09-29T22:12:07.648 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000387189586a1aba46-0065174bb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.648 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-12 2023-09-29T22:12:07.649 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:07.649 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:07.649 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.649 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:07.649 DEBUG:boto:Data: 2023-09-29T22:12:07.650 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.650 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.650 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.650 DEBUG:boto:Params: {} 2023-09-29T22:12:07.650 DEBUG:boto:Token: None 2023-09-29T22:12:07.651 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/ 2023-09-29T22:12:07.651 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+U1lghSAx0TJHBPUSWAO675Z2wE= 2023-09-29T22:12:07.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+U1lghSAx0TJHBPUSWAO675Z2wE=', 'Content-Length': '0'} 2023-09-29T22:12:07.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e804fd519f1c590-0065174bb7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.672 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:07.672 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:07.672 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.672 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:07.673 DEBUG:boto:Data: 2023-09-29T22:12:07.673 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.673 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.673 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.673 DEBUG:boto:Params: {} 2023-09-29T22:12:07.673 DEBUG:boto:Token: None 2023-09-29T22:12:07.674 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/ 2023-09-29T22:12:07.674 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo= 2023-09-29T22:12:07.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo=', 'Content-Length': '0'} 2023-09-29T22:12:07.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008243d7e3df18236b-0065174bb7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.677 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:07.677 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:07.678 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.678 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:07.678 DEBUG:boto:Data: 2023-09-29T22:12:07.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.678 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.678 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.679 DEBUG:boto:Params: {} 2023-09-29T22:12:07.679 DEBUG:boto:Token: None 2023-09-29T22:12:07.679 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/myobj 2023-09-29T22:12:07.679 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DFY68rCt5G1vT5x/p6oECXVI7II= 2023-09-29T22:12:07.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DFY68rCt5G1vT5x/p6oECXVI7II='} 2023-09-29T22:12:07.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a9499bef9a1ed98-0065174bb7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.688 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:07.688 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:07.688 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.688 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:07.689 DEBUG:boto:Data: 2023-09-29T22:12:07.689 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.689 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.689 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.689 DEBUG:boto:Params: {} 2023-09-29T22:12:07.689 DEBUG:boto:Token: None 2023-09-29T22:12:07.689 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/ 2023-09-29T22:12:07.689 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo= 2023-09-29T22:12:07.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo=', 'Content-Length': '0'} 2023-09-29T22:12:07.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ce3a75858720ef5-0065174bb7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.691 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:07.692 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:07.692 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.692 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:07.692 DEBUG:boto:Data: 2023-09-29T22:12:07.693 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.693 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.693 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.693 DEBUG:boto:Params: {} 2023-09-29T22:12:07.693 DEBUG:boto:Token: None 2023-09-29T22:12:07.693 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:07.693 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QITFQqsSI8VBb9j5JgbL1d0Wv9M= 2023-09-29T22:12:07.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QITFQqsSI8VBb9j5JgbL1d0Wv9M='} 2023-09-29T22:12:07.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1c098ba4fd606de-0065174bb7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.701 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:07.701 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:07.701 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.702 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:07.702 DEBUG:boto:Data: 2023-09-29T22:12:07.702 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.702 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.702 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.702 DEBUG:boto:Params: {} 2023-09-29T22:12:07.702 DEBUG:boto:Token: None 2023-09-29T22:12:07.702 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/ 2023-09-29T22:12:07.702 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo= 2023-09-29T22:12:07.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo=', 'Content-Length': '0'} 2023-09-29T22:12:07.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b13335086cf53c7-0065174bb7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.705 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:07.705 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:07.705 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.706 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:07.706 DEBUG:boto:Data: 2023-09-29T22:12:07.706 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.706 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.706 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.706 DEBUG:boto:Params: {} 2023-09-29T22:12:07.706 DEBUG:boto:Token: None 2023-09-29T22:12:07.706 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/%3A 2023-09-29T22:12:07.706 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b3vkLIxV4+3Rii3ZQSYfCzokvNI= 2023-09-29T22:12:07.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b3vkLIxV4+3Rii3ZQSYfCzokvNI='} 2023-09-29T22:12:07.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004dbbc0d10bf205be-0065174bb7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.751 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:07.751 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:07.751 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.751 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:07.751 DEBUG:boto:Data: 2023-09-29T22:12:07.751 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.751 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.751 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.752 DEBUG:boto:Params: {} 2023-09-29T22:12:07.752 DEBUG:boto:Token: None 2023-09-29T22:12:07.752 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/ 2023-09-29T22:12:07.752 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo= 2023-09-29T22:12:07.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:24nbhuh/QENoAJUp667/jS9HBlo=', 'Content-Length': '0'} 2023-09-29T22:12:07.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002014db07f21ccade-0065174bb7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.756 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:07.756 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:07.756 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.756 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:07.756 DEBUG:boto:Data: 2023-09-29T22:12:07.756 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.756 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:07.756 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.756 DEBUG:boto:Params: {} 2023-09-29T22:12:07.757 DEBUG:boto:Token: None 2023-09-29T22:12:07.757 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-10/%26 2023-09-29T22:12:07.757 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MbN53R/CtKmtL2Q1rGY9VbEH+8I= 2023-09-29T22:12:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MbN53R/CtKmtL2Q1rGY9VbEH+8I='} 2023-09-29T22:12:07.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fda45f5fee55880e-0065174bb7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.769 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:07.770 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:07.770 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.770 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:07.770 DEBUG:boto:Data: 2023-09-29T22:12:07.770 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.770 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.770 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.770 DEBUG:boto:Params: {} 2023-09-29T22:12:07.770 DEBUG:boto:Token: None 2023-09-29T22:12:07.771 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/ 2023-09-29T22:12:07.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo= 2023-09-29T22:12:07.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo=', 'Content-Length': '0'} 2023-09-29T22:12:07.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f77d2c2617ca04fa-0065174bb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.773 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:07.773 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:07.773 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.773 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:07.774 DEBUG:boto:Data: 2023-09-29T22:12:07.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.774 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.774 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.774 DEBUG:boto:Params: {} 2023-09-29T22:12:07.774 DEBUG:boto:Token: None 2023-09-29T22:12:07.774 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/myobj 2023-09-29T22:12:07.774 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UPZbNeYjkxFiJz3PBcI6VLdFjp0= 2023-09-29T22:12:07.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UPZbNeYjkxFiJz3PBcI6VLdFjp0='} 2023-09-29T22:12:07.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005aa5ae14fbfa74a5-0065174bb7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.785 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:07.785 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:07.785 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.785 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:07.785 DEBUG:boto:Data: 2023-09-29T22:12:07.785 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.785 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.785 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.785 DEBUG:boto:Params: {} 2023-09-29T22:12:07.785 DEBUG:boto:Token: None 2023-09-29T22:12:07.786 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/ 2023-09-29T22:12:07.786 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo= 2023-09-29T22:12:07.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo=', 'Content-Length': '0'} 2023-09-29T22:12:07.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f3feeb27fa3abc-0065174bb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.788 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:07.788 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:07.788 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.788 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:07.788 DEBUG:boto:Data: 2023-09-29T22:12:07.789 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.789 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.789 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.789 DEBUG:boto:Params: {} 2023-09-29T22:12:07.789 DEBUG:boto:Token: None 2023-09-29T22:12:07.789 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:07.789 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7tO/3yLs4IUS0iCSL9wfCg0Wpzw= 2023-09-29T22:12:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7tO/3yLs4IUS0iCSL9wfCg0Wpzw='} 2023-09-29T22:12:07.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ded7844eb30d5c3-0065174bb7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.801 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:07.801 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:07.801 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.801 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:07.801 DEBUG:boto:Data: 2023-09-29T22:12:07.802 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.802 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.802 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.802 DEBUG:boto:Params: {} 2023-09-29T22:12:07.802 DEBUG:boto:Token: None 2023-09-29T22:12:07.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/ 2023-09-29T22:12:07.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo= 2023-09-29T22:12:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo=', 'Content-Length': '0'} 2023-09-29T22:12:07.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41f2e15c59bfe6e-0065174bb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.804 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:07.804 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:07.805 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.805 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:07.805 DEBUG:boto:Data: 2023-09-29T22:12:07.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.805 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.805 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.805 DEBUG:boto:Params: {} 2023-09-29T22:12:07.805 DEBUG:boto:Token: None 2023-09-29T22:12:07.805 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/%3A 2023-09-29T22:12:07.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WVwSKqS/s9+FRRlp3GSs9pkPWZU= 2023-09-29T22:12:07.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WVwSKqS/s9+FRRlp3GSs9pkPWZU='} 2023-09-29T22:12:07.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006151407035779c70-0065174bb7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.851 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:07.851 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:07.851 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.851 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:07.851 DEBUG:boto:Data: 2023-09-29T22:12:07.851 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.851 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.851 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.851 DEBUG:boto:Params: {} 2023-09-29T22:12:07.852 DEBUG:boto:Token: None 2023-09-29T22:12:07.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/ 2023-09-29T22:12:07.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo= 2023-09-29T22:12:07.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aCl8S8CrHkgONqyz03J/vNKXPpo=', 'Content-Length': '0'} 2023-09-29T22:12:07.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c92aa334f7daab27-0065174bb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.854 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:07.854 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:07.854 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.854 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:07.855 DEBUG:boto:Data: 2023-09-29T22:12:07.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.855 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:07.855 DEBUG:boto:Port: 8000 2023-09-29T22:12:07.855 DEBUG:boto:Params: {} 2023-09-29T22:12:07.855 DEBUG:boto:Token: None 2023-09-29T22:12:07.855 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-11/%26 2023-09-29T22:12:07.855 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XO2PWRnZa+L3uTCCQuhbYiBQRT4= 2023-09-29T22:12:07.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XO2PWRnZa+L3uTCCQuhbYiBQRT4='} 2023-09-29T22:12:07.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f819eca87b9c72d4-0065174bb7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.862 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:07.862 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:07.862 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.862 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:07.862 DEBUG:boto:Data: 2023-09-29T22:12:07.863 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.863 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.863 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.863 DEBUG:boto:Params: {} 2023-09-29T22:12:07.863 DEBUG:boto:Token: None 2023-09-29T22:12:07.863 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/ 2023-09-29T22:12:07.863 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8= 2023-09-29T22:12:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8=', 'Content-Length': '0'} 2023-09-29T22:12:07.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a60e683ddf8451cb-0065174bb7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.866 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:07.866 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:07.866 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.866 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:07.866 DEBUG:boto:Data: 2023-09-29T22:12:07.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.866 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.866 DEBUG:boto:Params: {} 2023-09-29T22:12:07.867 DEBUG:boto:Token: None 2023-09-29T22:12:07.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/myobj 2023-09-29T22:12:07.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QJfHuUm1xPWUw4dWod9eqhFfWo8= 2023-09-29T22:12:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QJfHuUm1xPWUw4dWod9eqhFfWo8='} 2023-09-29T22:12:07.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4759a5857fcad60-0065174bb7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.884 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:07.884 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:07.884 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.884 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:07.884 DEBUG:boto:Data: 2023-09-29T22:12:07.884 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.885 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.885 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.885 DEBUG:boto:Params: {} 2023-09-29T22:12:07.885 DEBUG:boto:Token: None 2023-09-29T22:12:07.885 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/ 2023-09-29T22:12:07.885 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8= 2023-09-29T22:12:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8=', 'Content-Length': '0'} 2023-09-29T22:12:07.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef87d21dbdcb9cdb-0065174bb7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.887 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:07.888 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:07.888 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.888 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:07.888 DEBUG:boto:Data: 2023-09-29T22:12:07.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.888 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.888 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.888 DEBUG:boto:Params: {} 2023-09-29T22:12:07.888 DEBUG:boto:Token: None 2023-09-29T22:12:07.889 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:07.889 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0BaPOvE19JPBtfb5COKEVpPbZzk= 2023-09-29T22:12:07.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0BaPOvE19JPBtfb5COKEVpPbZzk='} 2023-09-29T22:12:07.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000afca8d8f77026a5-0065174bb7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.902 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:07.902 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:07.902 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.903 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:07.903 DEBUG:boto:Data: 2023-09-29T22:12:07.903 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.903 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.903 DEBUG:boto:Params: {} 2023-09-29T22:12:07.903 DEBUG:boto:Token: None 2023-09-29T22:12:07.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/ 2023-09-29T22:12:07.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8= 2023-09-29T22:12:07.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8=', 'Content-Length': '0'} 2023-09-29T22:12:07.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693f55b8d52de507-0065174bb7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.906 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:07.906 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:07.906 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.907 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:07.907 DEBUG:boto:Data: 2023-09-29T22:12:07.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.907 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.907 DEBUG:boto:Params: {} 2023-09-29T22:12:07.907 DEBUG:boto:Token: None 2023-09-29T22:12:07.907 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/%3A 2023-09-29T22:12:07.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vCaM4npoODTb7pg0yuInllDIYXI= 2023-09-29T22:12:07.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vCaM4npoODTb7pg0yuInllDIYXI='} 2023-09-29T22:12:07.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0285978f1763e64-0065174bb7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.951 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:07.951 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:07.951 DEBUG:boto:Method: HEAD 2023-09-29T22:12:07.951 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:07.951 DEBUG:boto:Data: 2023-09-29T22:12:07.952 DEBUG:boto:Headers: {} 2023-09-29T22:12:07.952 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.952 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.952 DEBUG:boto:Params: {} 2023-09-29T22:12:07.952 DEBUG:boto:Token: None 2023-09-29T22:12:07.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/ 2023-09-29T22:12:07.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8= 2023-09-29T22:12:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vLp/M/Fu9Q1SrrCjWwnaI3wPnd8=', 'Content-Length': '0'} 2023-09-29T22:12:07.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c090e7e1d08ec1b1-0065174bb7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.955 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:07.955 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:07.955 DEBUG:boto:Method: PUT 2023-09-29T22:12:07.955 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:07.955 DEBUG:boto:Data: 2023-09-29T22:12:07.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:07.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:07.956 DEBUG:boto:Port: 8001 2023-09-29T22:12:07.956 DEBUG:boto:Params: {} 2023-09-29T22:12:07.956 DEBUG:boto:Token: None 2023-09-29T22:12:07.956 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:07 GMT /dmuhsu-12/%26 2023-09-29T22:12:07.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nsxEIizXN/Cg9LR01iHnCqT0cxk= 2023-09-29T22:12:07.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nsxEIizXN/Cg9LR01iHnCqT0cxk='} 2023-09-29T22:12:07.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d612c6cdfd2c4d9e-0065174bb7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:07.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:07.961 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:08.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:08.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:08.034 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:12:08.045 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:12:08.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:08.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025527.664270_68.1", 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:12:07.664270Z" 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:08.047 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:12:08.048 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:08.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:12:08.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:08.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:08.049 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:08.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:08.098 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:08.129 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:08.129+0000 7f75ea630880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:08.129 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:08.130+0000 7f75ea630880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:08.141 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:12:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:12:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:12:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:12:08Z" 2023-09-29T22:12:08.148 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:12:08.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:08Z" } 2023-09-29T22:12:08.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:08.149 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:08.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:08.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696025527.664270_68.1 target= 2023-09-29T22:12:13.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:13.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:13.151 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:13.204 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:13.205 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:13.241 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:13.240+0000 7fdc5a520880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:13.241 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:13.241+0000 7fdc5a520880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:13.255 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:12:13.255 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:12:13.255 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:13.256 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:13.257 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:13.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025527.664270_68.1", 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:12:07.664270Z", 2023-09-29T22:12:13.259 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:13.260 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:12:13.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:13.262 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:12:13Z" 2023-09-29T22:12:13.262 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:12:13.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025527.664270_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:07.664270Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:13Z" } 2023-09-29T22:12:13.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025527.664270_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:07.664270Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:13.263 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:13.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:13.263 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:12:13.263 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:12:13.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:13.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:13.264 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:13.318 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:13.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:13.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:13.349+0000 7f0c46926880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:13.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:13.350+0000 7f0c46926880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:13.361 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:12:13.361 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:13.362 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:13.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025527.664270_68.1", 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:13.365 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:12:07.664270Z", 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:13.366 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:12:13.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:13.368 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:12:13Z" 2023-09-29T22:12:13.368 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:12:13.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025527.664270_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:07.664270Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:13Z" } 2023-09-29T22:12:13.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025527.664270_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:07.664270Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:13.369 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:13.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025527.664270_68.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:13.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:12:13.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:13.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:13.370 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:13.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:13.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:13.456 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:13.456+0000 7fbdcbc44880 1 waiting to reach incremental sync.. 2023-09-29T22:12:14.313 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:12:14.313+0000 7f303f796640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:12:14.355 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:12:14.355+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:12:18.458 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:18.458+0000 7fbdcbc44880 1 bucket sync caught up with source: 2023-09-29T22:12:18.459 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.4.6, , 00000000001.7.6, , , , 00000000001.9.6, 00000000001.10.6, , ] 2023-09-29T22:12:18.459 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.4.6, , 00000000001.7.6, , , , 00000000001.9.6, 00000000001.10.6, , ] 2023-09-29T22:12:18.459 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:18.458+0000 7fbdcbc44880 0 bucket checkpoint complete 2023-09-29T22:12:18.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-10 zones={test-zone2, test-zone1} 2023-09-29T22:12:18.474 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.474 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.474 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.475 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:18.475 DEBUG:boto:Data: 2023-09-29T22:12:18.475 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.475 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.475 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.475 DEBUG:boto:Params: {} 2023-09-29T22:12:18.475 DEBUG:boto:Token: None 2023-09-29T22:12:18.475 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a86nBN/TKQ3/yDL0AJtNhnXY7CI= 2023-09-29T22:12:18.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a86nBN/TKQ3/yDL0AJtNhnXY7CI=', 'Content-Length': '0'} 2023-09-29T22:12:18.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084e28db35801e296-0065174bc2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.478 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.478 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.479 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.479 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:18.479 DEBUG:boto:Data: 2023-09-29T22:12:18.479 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.479 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.479 DEBUG:boto:Params: {} 2023-09-29T22:12:18.479 DEBUG:boto:Token: None 2023-09-29T22:12:18.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a86nBN/TKQ3/yDL0AJtNhnXY7CI= 2023-09-29T22:12:18.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a86nBN/TKQ3/yDL0AJtNhnXY7CI=', 'Content-Length': '0'} 2023-09-29T22:12:18.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9722e8008d0c300-0065174bc2-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.482 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:18.483 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.483 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.483 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.483 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.483 DEBUG:boto:Method: GET 2023-09-29T22:12:18.483 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.483 DEBUG:boto:Data: 2023-09-29T22:12:18.483 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.484 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.484 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.484 DEBUG:boto:Params: {} 2023-09-29T22:12:18.484 DEBUG:boto:Token: None 2023-09-29T22:12:18.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/?versions 2023-09-29T22:12:18.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo= 2023-09-29T22:12:18.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo=', 'Content-Length': '0'} 2023-09-29T22:12:18.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f8be4fd14d6a60-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.486 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.488 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:18.488 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:18.488 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:18.489 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:18.489 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:18.489 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.489 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.489 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.489 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.489 DEBUG:boto:Method: GET 2023-09-29T22:12:18.489 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.489 DEBUG:boto:Data: 2023-09-29T22:12:18.490 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.490 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.490 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.490 DEBUG:boto:Params: {} 2023-09-29T22:12:18.490 DEBUG:boto:Token: None 2023-09-29T22:12:18.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/?versions 2023-09-29T22:12:18.490 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo= 2023-09-29T22:12:18.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo=', 'Content-Length': '0'} 2023-09-29T22:12:18.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008783955b8179557e-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.493 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.493 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:18.493 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:18.493 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:18.494 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:18.494 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.494 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.494 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.494 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.494 DEBUG:boto:Method: GET 2023-09-29T22:12:18.494 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.494 DEBUG:boto:Data: 2023-09-29T22:12:18.495 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.495 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.495 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.495 DEBUG:boto:Params: {} 2023-09-29T22:12:18.495 DEBUG:boto:Token: None 2023-09-29T22:12:18.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/?versions 2023-09-29T22:12:18.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo= 2023-09-29T22:12:18.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo=', 'Content-Length': '0'} 2023-09-29T22:12:18.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b118410e9a58f63-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.498 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.498 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.499 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.499 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.499 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.499 DEBUG:boto:Method: GET 2023-09-29T22:12:18.499 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:18.499 DEBUG:boto:Data: 2023-09-29T22:12:18.499 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.499 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.499 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.499 DEBUG:boto:Params: {} 2023-09-29T22:12:18.500 DEBUG:boto:Token: None 2023-09-29T22:12:18.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/?versions 2023-09-29T22:12:18.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo= 2023-09-29T22:12:18.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQkcMaI2AJCyBAXb3B4+2BZo5mo=', 'Content-Length': '0'} 2023-09-29T22:12:18.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f776f4696778f64-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.502 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:18.503 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.503 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.503 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.503 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.504 DEBUG:boto:Method: GET 2023-09-29T22:12:18.504 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.504 DEBUG:boto:Data: 2023-09-29T22:12:18.504 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.504 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.504 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.504 DEBUG:boto:Params: {} 2023-09-29T22:12:18.504 DEBUG:boto:Token: None 2023-09-29T22:12:18.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE= 2023-09-29T22:12:18.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE=', 'Content-Length': '0'} 2023-09-29T22:12:18.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4145154338129f4-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.507 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.507 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.507 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.507 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.508 DEBUG:boto:Method: GET 2023-09-29T22:12:18.508 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.508 DEBUG:boto:Data: 2023-09-29T22:12:18.508 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.508 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.508 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.508 DEBUG:boto:Params: {} 2023-09-29T22:12:18.508 DEBUG:boto:Token: None 2023-09-29T22:12:18.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE= 2023-09-29T22:12:18.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE=', 'Content-Length': '0'} 2023-09-29T22:12:18.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b26f0e0c685218a-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.554 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.555 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.555 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.555 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.555 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.555 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.555 DEBUG:boto:Data: 2023-09-29T22:12:18.555 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.555 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.555 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.555 DEBUG:boto:Params: {} 2023-09-29T22:12:18.556 DEBUG:boto:Token: None 2023-09-29T22:12:18.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.556 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SRIjXZlyRi62hxuOPk7660AzGS4= 2023-09-29T22:12:18.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SRIjXZlyRi62hxuOPk7660AzGS4=', 'Content-Length': '0'} 2023-09-29T22:12:18.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdb09570318693c8-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.558 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.558 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.559 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.559 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.559 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.559 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.559 DEBUG:boto:Data: 2023-09-29T22:12:18.559 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.559 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.559 DEBUG:boto:Params: {} 2023-09-29T22:12:18.560 DEBUG:boto:Token: None 2023-09-29T22:12:18.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SRIjXZlyRi62hxuOPk7660AzGS4= 2023-09-29T22:12:18.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SRIjXZlyRi62hxuOPk7660AzGS4=', 'Content-Length': '0'} 2023-09-29T22:12:18.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc1e2fd8533972d3-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:18.562 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.562 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.563 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.563 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.563 DEBUG:boto:Method: GET 2023-09-29T22:12:18.563 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.563 DEBUG:boto:Data: 2023-09-29T22:12:18.563 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.563 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.563 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.563 DEBUG:boto:Params: {} 2023-09-29T22:12:18.564 DEBUG:boto:Token: None 2023-09-29T22:12:18.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE= 2023-09-29T22:12:18.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE=', 'Content-Length': '0'} 2023-09-29T22:12:18.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001082da363b7d0831-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.566 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.566 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.566 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.566 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.566 DEBUG:boto:Method: GET 2023-09-29T22:12:18.566 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.567 DEBUG:boto:Data: 2023-09-29T22:12:18.567 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.567 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.567 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.567 DEBUG:boto:Params: {} 2023-09-29T22:12:18.567 DEBUG:boto:Token: None 2023-09-29T22:12:18.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:18.567 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE= 2023-09-29T22:12:18.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zfL/MbjE2Tpd13xbNtJqkxScsQE=', 'Content-Length': '0'} 2023-09-29T22:12:18.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b031cf7c6bb602c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.569 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.569 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.569 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.570 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.570 DEBUG:boto:Method: GET 2023-09-29T22:12:18.570 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.570 DEBUG:boto:Data: 2023-09-29T22:12:18.570 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.570 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.570 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.570 DEBUG:boto:Params: {} 2023-09-29T22:12:18.570 DEBUG:boto:Token: None 2023-09-29T22:12:18.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064283825085cd8ac-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.573 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.573 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.573 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.574 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.574 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.574 DEBUG:boto:Method: GET 2023-09-29T22:12:18.574 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.574 DEBUG:boto:Data: 2023-09-29T22:12:18.574 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.574 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.574 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.574 DEBUG:boto:Params: {} 2023-09-29T22:12:18.574 DEBUG:boto:Token: None 2023-09-29T22:12:18.575 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.575 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b2e4e884dad751-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.577 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.577 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.577 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.577 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.578 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:18.578 DEBUG:boto:Data: 2023-09-29T22:12:18.578 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.578 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.578 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.578 DEBUG:boto:Params: {} 2023-09-29T22:12:18.578 DEBUG:boto:Token: None 2023-09-29T22:12:18.578 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26 2023-09-29T22:12:18.578 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n7RtNzjqiwfwLjwc2NFMa0sb+1c= 2023-09-29T22:12:18.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n7RtNzjqiwfwLjwc2NFMa0sb+1c=', 'Content-Length': '0'} 2023-09-29T22:12:18.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000326b308eecd5ed59-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.580 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.580 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.581 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.581 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:18.581 DEBUG:boto:Data: 2023-09-29T22:12:18.581 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.581 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.581 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.581 DEBUG:boto:Params: {} 2023-09-29T22:12:18.581 DEBUG:boto:Token: None 2023-09-29T22:12:18.581 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26 2023-09-29T22:12:18.581 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n7RtNzjqiwfwLjwc2NFMa0sb+1c= 2023-09-29T22:12:18.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n7RtNzjqiwfwLjwc2NFMa0sb+1c=', 'Content-Length': '0'} 2023-09-29T22:12:18.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000164dabd8316765b-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:18.583 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.584 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.584 DEBUG:boto:Method: GET 2023-09-29T22:12:18.584 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:18.584 DEBUG:boto:Data: 2023-09-29T22:12:18.584 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.584 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.584 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.584 DEBUG:boto:Params: {} 2023-09-29T22:12:18.584 DEBUG:boto:Token: None 2023-09-29T22:12:18.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26 2023-09-29T22:12:18.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R5+iMNteFl3yUTatBGYC1613T1g= 2023-09-29T22:12:18.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R5+iMNteFl3yUTatBGYC1613T1g=', 'Content-Length': '0'} 2023-09-29T22:12:18.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf1504bb1d8273f4-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.587 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:18.587 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:18.587 DEBUG:boto:Method: GET 2023-09-29T22:12:18.587 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:18.587 DEBUG:boto:Data: 2023-09-29T22:12:18.587 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.587 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.587 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.587 DEBUG:boto:Params: {} 2023-09-29T22:12:18.587 DEBUG:boto:Token: None 2023-09-29T22:12:18.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%26 2023-09-29T22:12:18.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R5+iMNteFl3yUTatBGYC1613T1g= 2023-09-29T22:12:18.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R5+iMNteFl3yUTatBGYC1613T1g=', 'Content-Length': '0'} 2023-09-29T22:12:18.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000920dad1fb78d8af5-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.590 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.590 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.590 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.590 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.590 DEBUG:boto:Method: GET 2023-09-29T22:12:18.590 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.591 DEBUG:boto:Data: 2023-09-29T22:12:18.591 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.591 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.591 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.591 DEBUG:boto:Params: {} 2023-09-29T22:12:18.591 DEBUG:boto:Token: None 2023-09-29T22:12:18.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dfffb50899926c7-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.634 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.635 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.635 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.635 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.635 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.635 DEBUG:boto:Method: GET 2023-09-29T22:12:18.635 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:18.635 DEBUG:boto:Data: 2023-09-29T22:12:18.635 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.636 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.636 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.636 DEBUG:boto:Params: {} 2023-09-29T22:12:18.636 DEBUG:boto:Token: None 2023-09-29T22:12:18.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004efc984516f0aa8f-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.639 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:18.639 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.639 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.639 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.640 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.640 DEBUG:boto:Method: GET 2023-09-29T22:12:18.640 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.640 DEBUG:boto:Data: 2023-09-29T22:12:18.640 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.640 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.640 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.640 DEBUG:boto:Params: {} 2023-09-29T22:12:18.640 DEBUG:boto:Token: None 2023-09-29T22:12:18.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.641 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA= 2023-09-29T22:12:18.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA=', 'Content-Length': '0'} 2023-09-29T22:12:18.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd81da07fa1e9202-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.643 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.643 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.643 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.643 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.643 DEBUG:boto:Method: GET 2023-09-29T22:12:18.643 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.643 DEBUG:boto:Data: 2023-09-29T22:12:18.643 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.643 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.644 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.644 DEBUG:boto:Params: {} 2023-09-29T22:12:18.644 DEBUG:boto:Token: None 2023-09-29T22:12:18.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA= 2023-09-29T22:12:18.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA=', 'Content-Length': '0'} 2023-09-29T22:12:18.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080f4a6c67811d70c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.686 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.686 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.687 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.687 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.687 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.687 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.687 DEBUG:boto:Data: 2023-09-29T22:12:18.687 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.687 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.687 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.687 DEBUG:boto:Params: {} 2023-09-29T22:12:18.687 DEBUG:boto:Token: None 2023-09-29T22:12:18.688 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.688 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ss+deA9EJU2pmgPcW9+kiBUeTZs= 2023-09-29T22:12:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ss+deA9EJU2pmgPcW9+kiBUeTZs=', 'Content-Length': '0'} 2023-09-29T22:12:18.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005abe5f09a5ad07c-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.690 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.690 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.690 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.690 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.690 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.690 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.690 DEBUG:boto:Data: 2023-09-29T22:12:18.690 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.690 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.691 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.691 DEBUG:boto:Params: {} 2023-09-29T22:12:18.691 DEBUG:boto:Token: None 2023-09-29T22:12:18.691 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ss+deA9EJU2pmgPcW9+kiBUeTZs= 2023-09-29T22:12:18.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ss+deA9EJU2pmgPcW9+kiBUeTZs=', 'Content-Length': '0'} 2023-09-29T22:12:18.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7ffde381eafae1c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:18.693 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.693 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.693 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.693 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.693 DEBUG:boto:Method: GET 2023-09-29T22:12:18.694 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.694 DEBUG:boto:Data: 2023-09-29T22:12:18.694 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.694 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.694 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.694 DEBUG:boto:Params: {} 2023-09-29T22:12:18.694 DEBUG:boto:Token: None 2023-09-29T22:12:18.694 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.694 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA= 2023-09-29T22:12:18.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA=', 'Content-Length': '0'} 2023-09-29T22:12:18.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c0fac070191aa3b-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.696 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.697 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.697 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.697 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.697 DEBUG:boto:Method: GET 2023-09-29T22:12:18.697 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.697 DEBUG:boto:Data: 2023-09-29T22:12:18.697 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.697 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.697 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.697 DEBUG:boto:Params: {} 2023-09-29T22:12:18.697 DEBUG:boto:Token: None 2023-09-29T22:12:18.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:18.698 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA= 2023-09-29T22:12:18.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t7OCwqPB3V54eHDVsWTUbd7UOcA=', 'Content-Length': '0'} 2023-09-29T22:12:18.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000baebef4543a68783-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.700 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.700 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.700 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.700 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.700 DEBUG:boto:Method: GET 2023-09-29T22:12:18.700 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.700 DEBUG:boto:Data: 2023-09-29T22:12:18.700 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.701 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.701 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.701 DEBUG:boto:Params: {} 2023-09-29T22:12:18.701 DEBUG:boto:Token: None 2023-09-29T22:12:18.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.701 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3134e6644b63411-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.704 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.704 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.704 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.704 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.704 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.704 DEBUG:boto:Method: GET 2023-09-29T22:12:18.704 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.705 DEBUG:boto:Data: 2023-09-29T22:12:18.705 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.705 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.705 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.705 DEBUG:boto:Params: {} 2023-09-29T22:12:18.705 DEBUG:boto:Token: None 2023-09-29T22:12:18.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.705 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0155c4f546c1f49-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.708 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.708 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.708 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.708 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.708 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:18.708 DEBUG:boto:Data: 2023-09-29T22:12:18.708 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.709 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.709 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.709 DEBUG:boto:Params: {} 2023-09-29T22:12:18.709 DEBUG:boto:Token: None 2023-09-29T22:12:18.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A 2023-09-29T22:12:18.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OpWC8KcW2dWW6vlgdtmqi1/brCQ= 2023-09-29T22:12:18.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OpWC8KcW2dWW6vlgdtmqi1/brCQ=', 'Content-Length': '0'} 2023-09-29T22:12:18.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa8f53c9455227e4-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.711 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.711 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.711 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.711 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:18.711 DEBUG:boto:Data: 2023-09-29T22:12:18.712 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.712 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.712 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.712 DEBUG:boto:Params: {} 2023-09-29T22:12:18.712 DEBUG:boto:Token: None 2023-09-29T22:12:18.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A 2023-09-29T22:12:18.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OpWC8KcW2dWW6vlgdtmqi1/brCQ= 2023-09-29T22:12:18.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OpWC8KcW2dWW6vlgdtmqi1/brCQ=', 'Content-Length': '0'} 2023-09-29T22:12:18.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6081d8a45793171-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:18.714 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.715 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.715 DEBUG:boto:Method: GET 2023-09-29T22:12:18.715 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:18.715 DEBUG:boto:Data: 2023-09-29T22:12:18.715 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.715 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.715 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.715 DEBUG:boto:Params: {} 2023-09-29T22:12:18.715 DEBUG:boto:Token: None 2023-09-29T22:12:18.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A 2023-09-29T22:12:18.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ioT9E7aAdZiWPvejQ6ZO+icLjHM= 2023-09-29T22:12:18.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ioT9E7aAdZiWPvejQ6ZO+icLjHM=', 'Content-Length': '0'} 2023-09-29T22:12:18.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000928609c371c03ed9-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.758 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:18.758 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:18.758 DEBUG:boto:Method: GET 2023-09-29T22:12:18.759 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:18.759 DEBUG:boto:Data: 2023-09-29T22:12:18.759 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.759 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.759 DEBUG:boto:Params: {} 2023-09-29T22:12:18.759 DEBUG:boto:Token: None 2023-09-29T22:12:18.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/%3A 2023-09-29T22:12:18.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ioT9E7aAdZiWPvejQ6ZO+icLjHM= 2023-09-29T22:12:18.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ioT9E7aAdZiWPvejQ6ZO+icLjHM=', 'Content-Length': '0'} 2023-09-29T22:12:18.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007935301c4100424f-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.762 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.762 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.762 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.762 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.762 DEBUG:boto:Method: GET 2023-09-29T22:12:18.762 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.762 DEBUG:boto:Data: 2023-09-29T22:12:18.762 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.763 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.763 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.763 DEBUG:boto:Params: {} 2023-09-29T22:12:18.763 DEBUG:boto:Token: None 2023-09-29T22:12:18.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.763 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda1d2461f64b22d-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.766 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.766 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.766 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.767 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.767 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.767 DEBUG:boto:Method: GET 2023-09-29T22:12:18.767 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:18.767 DEBUG:boto:Data: 2023-09-29T22:12:18.767 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.767 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.767 DEBUG:boto:Params: {} 2023-09-29T22:12:18.767 DEBUG:boto:Token: None 2023-09-29T22:12:18.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c0d5d8672bb47fd-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.770 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:18.771 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.771 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.771 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.771 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.771 DEBUG:boto:Method: GET 2023-09-29T22:12:18.771 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.771 DEBUG:boto:Data: 2023-09-29T22:12:18.771 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.771 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.772 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.772 DEBUG:boto:Params: {} 2023-09-29T22:12:18.772 DEBUG:boto:Token: None 2023-09-29T22:12:18.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I= 2023-09-29T22:12:18.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I=', 'Content-Length': '0'} 2023-09-29T22:12:18.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ea01214c3865a22-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.774 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.774 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.774 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.774 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.774 DEBUG:boto:Method: GET 2023-09-29T22:12:18.774 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.775 DEBUG:boto:Data: 2023-09-29T22:12:18.775 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.775 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.775 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.775 DEBUG:boto:Params: {} 2023-09-29T22:12:18.775 DEBUG:boto:Token: None 2023-09-29T22:12:18.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.775 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I= 2023-09-29T22:12:18.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I=', 'Content-Length': '0'} 2023-09-29T22:12:18.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e0c1b5aa1803c3f-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.822 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.822 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.823 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.823 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.823 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.823 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.823 DEBUG:boto:Data: 2023-09-29T22:12:18.823 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.823 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.823 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.823 DEBUG:boto:Params: {} 2023-09-29T22:12:18.823 DEBUG:boto:Token: None 2023-09-29T22:12:18.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mkt1fsnHlOoA9oNfisa+qjavOhI= 2023-09-29T22:12:18.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mkt1fsnHlOoA9oNfisa+qjavOhI=', 'Content-Length': '0'} 2023-09-29T22:12:18.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3e71726a1a819f0-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.826 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.826 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.826 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.826 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.826 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.826 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.826 DEBUG:boto:Data: 2023-09-29T22:12:18.826 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.826 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.826 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.827 DEBUG:boto:Params: {} 2023-09-29T22:12:18.827 DEBUG:boto:Token: None 2023-09-29T22:12:18.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mkt1fsnHlOoA9oNfisa+qjavOhI= 2023-09-29T22:12:18.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mkt1fsnHlOoA9oNfisa+qjavOhI=', 'Content-Length': '0'} 2023-09-29T22:12:18.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdec9b8f1a02880c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:18.829 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.829 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.829 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.829 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.829 DEBUG:boto:Method: GET 2023-09-29T22:12:18.830 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.830 DEBUG:boto:Data: 2023-09-29T22:12:18.830 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.830 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.830 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.830 DEBUG:boto:Params: {} 2023-09-29T22:12:18.830 DEBUG:boto:Token: None 2023-09-29T22:12:18.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.830 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I= 2023-09-29T22:12:18.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I=', 'Content-Length': '0'} 2023-09-29T22:12:18.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db9fabf0312736c9-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.832 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.832 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.832 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.833 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.833 DEBUG:boto:Method: GET 2023-09-29T22:12:18.833 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.833 DEBUG:boto:Data: 2023-09-29T22:12:18.833 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.833 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.833 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.833 DEBUG:boto:Params: {} 2023-09-29T22:12:18.833 DEBUG:boto:Token: None 2023-09-29T22:12:18.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:18.834 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I= 2023-09-29T22:12:18.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3f+Yclk23rIuLrBa/giTmkyE3I=', 'Content-Length': '0'} 2023-09-29T22:12:18.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000582c5fbfb1d44e8a-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.836 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.836 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.836 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.836 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.836 DEBUG:boto:Method: GET 2023-09-29T22:12:18.836 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.836 DEBUG:boto:Data: 2023-09-29T22:12:18.836 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.836 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.836 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.837 DEBUG:boto:Params: {} 2023-09-29T22:12:18.837 DEBUG:boto:Token: None 2023-09-29T22:12:18.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.837 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081af3880a6926735-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.839 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.839 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.840 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.840 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.840 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.840 DEBUG:boto:Method: GET 2023-09-29T22:12:18.840 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.840 DEBUG:boto:Data: 2023-09-29T22:12:18.840 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.840 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.840 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.840 DEBUG:boto:Params: {} 2023-09-29T22:12:18.841 DEBUG:boto:Token: None 2023-09-29T22:12:18.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.841 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd4e58744ff6407-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.843 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.843 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.844 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.844 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.844 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:18.844 DEBUG:boto:Data: 2023-09-29T22:12:18.844 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.844 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.844 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.844 DEBUG:boto:Params: {} 2023-09-29T22:12:18.844 DEBUG:boto:Token: None 2023-09-29T22:12:18.845 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:18.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zOAjTt64d+Dx+YMYroGHTWtcZtU= 2023-09-29T22:12:18.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zOAjTt64d+Dx+YMYroGHTWtcZtU=', 'Content-Length': '0'} 2023-09-29T22:12:18.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000668f3d98f1f1198-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.847 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.847 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.847 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.847 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:18.847 DEBUG:boto:Data: 2023-09-29T22:12:18.847 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.847 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.847 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.847 DEBUG:boto:Params: {} 2023-09-29T22:12:18.848 DEBUG:boto:Token: None 2023-09-29T22:12:18.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:18.848 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zOAjTt64d+Dx+YMYroGHTWtcZtU= 2023-09-29T22:12:18.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zOAjTt64d+Dx+YMYroGHTWtcZtU=', 'Content-Length': '0'} 2023-09-29T22:12:18.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f9b0706576aba39-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:18.850 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.850 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.850 DEBUG:boto:Method: GET 2023-09-29T22:12:18.850 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:18.850 DEBUG:boto:Data: 2023-09-29T22:12:18.850 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.850 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.851 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.851 DEBUG:boto:Params: {} 2023-09-29T22:12:18.851 DEBUG:boto:Token: None 2023-09-29T22:12:18.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:18.851 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0jgMJSEBnwQIvTVN9VM7whq0jDk= 2023-09-29T22:12:18.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0jgMJSEBnwQIvTVN9VM7whq0jDk=', 'Content-Length': '0'} 2023-09-29T22:12:18.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f19f30a4cdec48ac-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.894 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.895 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:18.895 DEBUG:boto:Method: GET 2023-09-29T22:12:18.895 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:18.895 DEBUG:boto:Data: 2023-09-29T22:12:18.895 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.895 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.895 DEBUG:boto:Params: {} 2023-09-29T22:12:18.895 DEBUG:boto:Token: None 2023-09-29T22:12:18.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:18.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0jgMJSEBnwQIvTVN9VM7whq0jDk= 2023-09-29T22:12:18.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0jgMJSEBnwQIvTVN9VM7whq0jDk=', 'Content-Length': '0'} 2023-09-29T22:12:18.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043d1a240e0a1344d-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.898 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.898 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.898 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.898 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.899 DEBUG:boto:Method: GET 2023-09-29T22:12:18.899 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.899 DEBUG:boto:Data: 2023-09-29T22:12:18.899 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.899 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.899 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.899 DEBUG:boto:Params: {} 2023-09-29T22:12:18.899 DEBUG:boto:Token: None 2023-09-29T22:12:18.899 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa379bf8c914343-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.902 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.902 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.902 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.903 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.903 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.903 DEBUG:boto:Method: GET 2023-09-29T22:12:18.903 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:18.903 DEBUG:boto:Data: 2023-09-29T22:12:18.903 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.903 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.903 DEBUG:boto:Params: {} 2023-09-29T22:12:18.903 DEBUG:boto:Token: None 2023-09-29T22:12:18.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007321c7e71110bc1c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.906 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:18.907 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.907 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.907 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.907 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.907 DEBUG:boto:Method: GET 2023-09-29T22:12:18.907 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.907 DEBUG:boto:Data: 2023-09-29T22:12:18.907 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.908 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.908 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.908 DEBUG:boto:Params: {} 2023-09-29T22:12:18.908 DEBUG:boto:Token: None 2023-09-29T22:12:18.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4= 2023-09-29T22:12:18.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4=', 'Content-Length': '0'} 2023-09-29T22:12:18.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d45e684332a81856-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.910 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.910 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.910 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.910 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.911 DEBUG:boto:Method: GET 2023-09-29T22:12:18.911 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.911 DEBUG:boto:Data: 2023-09-29T22:12:18.911 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.911 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.911 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.911 DEBUG:boto:Params: {} 2023-09-29T22:12:18.911 DEBUG:boto:Token: None 2023-09-29T22:12:18.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.911 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4= 2023-09-29T22:12:18.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4=', 'Content-Length': '0'} 2023-09-29T22:12:18.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e431a18a5c060c17-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.914 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.914 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.914 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.914 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.914 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.914 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.914 DEBUG:boto:Data: 2023-09-29T22:12:18.914 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.915 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.915 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.915 DEBUG:boto:Params: {} 2023-09-29T22:12:18.915 DEBUG:boto:Token: None 2023-09-29T22:12:18.915 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.915 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ts2Ji/K2NEr9ufX0MW7o4F+YSow= 2023-09-29T22:12:18.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ts2Ji/K2NEr9ufX0MW7o4F+YSow=', 'Content-Length': '0'} 2023-09-29T22:12:18.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003edf07d24d79be22-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.917 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.917 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.917 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.917 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.918 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.918 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.918 DEBUG:boto:Data: 2023-09-29T22:12:18.918 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.918 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.918 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.918 DEBUG:boto:Params: {} 2023-09-29T22:12:18.918 DEBUG:boto:Token: None 2023-09-29T22:12:18.918 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.918 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ts2Ji/K2NEr9ufX0MW7o4F+YSow= 2023-09-29T22:12:18.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ts2Ji/K2NEr9ufX0MW7o4F+YSow=', 'Content-Length': '0'} 2023-09-29T22:12:18.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c10dca3fdc97ae8-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:18.920 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.921 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.921 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.921 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.921 DEBUG:boto:Method: GET 2023-09-29T22:12:18.921 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.921 DEBUG:boto:Data: 2023-09-29T22:12:18.921 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.921 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.921 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.922 DEBUG:boto:Params: {} 2023-09-29T22:12:18.922 DEBUG:boto:Token: None 2023-09-29T22:12:18.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.922 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4= 2023-09-29T22:12:18.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4=', 'Content-Length': '0'} 2023-09-29T22:12:18.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a78a0fd6167992-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.924 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.924 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.924 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.924 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.924 DEBUG:boto:Method: GET 2023-09-29T22:12:18.925 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.925 DEBUG:boto:Data: 2023-09-29T22:12:18.925 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.925 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.925 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.925 DEBUG:boto:Params: {} 2023-09-29T22:12:18.925 DEBUG:boto:Token: None 2023-09-29T22:12:18.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:18.926 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4= 2023-09-29T22:12:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zQeWm4hKXqXvdL3B4AkWmj7A5X4=', 'Content-Length': '0'} 2023-09-29T22:12:18.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1176179e7a7023a-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.928 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.928 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.928 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.928 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.928 DEBUG:boto:Method: GET 2023-09-29T22:12:18.928 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.928 DEBUG:boto:Data: 2023-09-29T22:12:18.928 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.928 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.929 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.929 DEBUG:boto:Params: {} 2023-09-29T22:12:18.929 DEBUG:boto:Token: None 2023-09-29T22:12:18.929 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.929 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb870eded1ccdc2-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.932 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.932 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.933 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.933 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.933 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.933 DEBUG:boto:Method: GET 2023-09-29T22:12:18.933 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.933 DEBUG:boto:Data: 2023-09-29T22:12:18.934 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.934 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.934 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.934 DEBUG:boto:Params: {} 2023-09-29T22:12:18.934 DEBUG:boto:Token: None 2023-09-29T22:12:18.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.935 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b26f27c07965ab-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.938 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.938 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.938 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.938 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.938 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:18.939 DEBUG:boto:Data: 2023-09-29T22:12:18.939 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.939 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.939 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.939 DEBUG:boto:Params: {} 2023-09-29T22:12:18.939 DEBUG:boto:Token: None 2023-09-29T22:12:18.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj 2023-09-29T22:12:18.939 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DdQz4JUMJNZ+QNYGsCosKwq+PHM= 2023-09-29T22:12:18.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DdQz4JUMJNZ+QNYGsCosKwq+PHM=', 'Content-Length': '0'} 2023-09-29T22:12:18.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f985a25382b578b7-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.941 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.942 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.942 DEBUG:boto:Method: HEAD 2023-09-29T22:12:18.942 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:18.942 DEBUG:boto:Data: 2023-09-29T22:12:18.942 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.942 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.942 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.942 DEBUG:boto:Params: {} 2023-09-29T22:12:18.943 DEBUG:boto:Token: None 2023-09-29T22:12:18.943 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj 2023-09-29T22:12:18.943 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DdQz4JUMJNZ+QNYGsCosKwq+PHM= 2023-09-29T22:12:18.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DdQz4JUMJNZ+QNYGsCosKwq+PHM=', 'Content-Length': '0'} 2023-09-29T22:12:18.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8f607fd1d3876d0-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:18.945 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.945 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.945 DEBUG:boto:Method: GET 2023-09-29T22:12:18.946 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:18.946 DEBUG:boto:Data: 2023-09-29T22:12:18.946 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.946 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.946 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.946 DEBUG:boto:Params: {} 2023-09-29T22:12:18.946 DEBUG:boto:Token: None 2023-09-29T22:12:18.947 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj 2023-09-29T22:12:18.947 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lqppOOYnv+PJ08i0hXHcEkU8NsA= 2023-09-29T22:12:18.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lqppOOYnv+PJ08i0hXHcEkU8NsA=', 'Content-Length': '0'} 2023-09-29T22:12:18.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049c20bb31e154cfe-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.949 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:18.949 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:18.949 DEBUG:boto:Method: GET 2023-09-29T22:12:18.949 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:18.949 DEBUG:boto:Data: 2023-09-29T22:12:18.949 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.949 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.949 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.949 DEBUG:boto:Params: {} 2023-09-29T22:12:18.950 DEBUG:boto:Token: None 2023-09-29T22:12:18.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/myobj 2023-09-29T22:12:18.950 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lqppOOYnv+PJ08i0hXHcEkU8NsA= 2023-09-29T22:12:18.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lqppOOYnv+PJ08i0hXHcEkU8NsA=', 'Content-Length': '0'} 2023-09-29T22:12:18.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3e7c08ef8898ba6-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.952 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.952 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.953 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.953 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.953 DEBUG:boto:Method: GET 2023-09-29T22:12:18.953 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.953 DEBUG:boto:Data: 2023-09-29T22:12:18.953 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.953 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:18.953 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.953 DEBUG:boto:Params: {} 2023-09-29T22:12:18.953 DEBUG:boto:Token: None 2023-09-29T22:12:18.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.954 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53e9fa6cfcf1050-0065174bc2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.956 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.957 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:18.957 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:18.957 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.957 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.957 DEBUG:boto:Method: GET 2023-09-29T22:12:18.957 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:18.957 DEBUG:boto:Data: 2023-09-29T22:12:18.958 DEBUG:boto:Headers: {} 2023-09-29T22:12:18.958 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:18.958 DEBUG:boto:Port: 8000 2023-09-29T22:12:18.958 DEBUG:boto:Params: {} 2023-09-29T22:12:18.958 DEBUG:boto:Token: None 2023-09-29T22:12:18.958 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:18 GMT /dmuhsu-10/ 2023-09-29T22:12:18.958 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw= 2023-09-29T22:12:18.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jmppldv3vSz6x77h3n8A3UIXLnw=', 'Content-Length': '0'} 2023-09-29T22:12:18.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460a18d5e9a1fe4c-0065174bc2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:18.961 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:18.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-10 zones={test-zone2, test-zone1} 2023-09-29T22:12:18.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:18.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:18.962 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:19.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:19.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:19.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:19.053+0000 7efed7736880 1 waiting to reach incremental sync.. 2023-09-29T22:12:24.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:24.055+0000 7efed7736880 1 bucket sync caught up with source: 2023-09-29T22:12:24.055 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.4.6, , 00000000001.7.6, , , , 00000000001.9.6, 00000000001.10.6, , ] 2023-09-29T22:12:24.055 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.4.6, , 00000000001.7.6, , , , 00000000001.9.6, 00000000001.10.6, , ] 2023-09-29T22:12:24.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:24.055+0000 7efed7736880 0 bucket checkpoint complete 2023-09-29T22:12:24.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-10 zones={test-zone3, test-zone1} 2023-09-29T22:12:24.072 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.072 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.072 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.072 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:24.072 DEBUG:boto:Data: 2023-09-29T22:12:24.072 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.073 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.073 DEBUG:boto:Params: {} 2023-09-29T22:12:24.073 DEBUG:boto:Token: None 2023-09-29T22:12:24.073 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SikHsyqOtp4TV0g2cENvUbtrmEg= 2023-09-29T22:12:24.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SikHsyqOtp4TV0g2cENvUbtrmEg=', 'Content-Length': '0'} 2023-09-29T22:12:24.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8d2c803494b6ce0-0065174bc8-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.076 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.076 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.076 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.076 DEBUG:boto:Path: /dmuhsu-10/ 2023-09-29T22:12:24.076 DEBUG:boto:Data: 2023-09-29T22:12:24.076 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.076 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.076 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.077 DEBUG:boto:Params: {} 2023-09-29T22:12:24.077 DEBUG:boto:Token: None 2023-09-29T22:12:24.077 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.077 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SikHsyqOtp4TV0g2cENvUbtrmEg= 2023-09-29T22:12:24.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SikHsyqOtp4TV0g2cENvUbtrmEg=', 'Content-Length': '0'} 2023-09-29T22:12:24.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc154d1944a02a94-0065174bc8-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.079 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:24.080 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.080 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.080 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.080 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.080 DEBUG:boto:Method: GET 2023-09-29T22:12:24.080 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.080 DEBUG:boto:Data: 2023-09-29T22:12:24.080 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.080 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.080 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.081 DEBUG:boto:Params: {} 2023-09-29T22:12:24.081 DEBUG:boto:Token: None 2023-09-29T22:12:24.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/?versions 2023-09-29T22:12:24.081 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU= 2023-09-29T22:12:24.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU=', 'Content-Length': '0'} 2023-09-29T22:12:24.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f3133297167ca9c-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.083 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.084 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:24.084 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:24.084 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:24.084 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:24.084 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:24.084 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.084 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.085 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.085 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.085 DEBUG:boto:Method: GET 2023-09-29T22:12:24.085 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.085 DEBUG:boto:Data: 2023-09-29T22:12:24.085 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.085 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.085 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.085 DEBUG:boto:Params: {} 2023-09-29T22:12:24.085 DEBUG:boto:Token: None 2023-09-29T22:12:24.086 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/?versions 2023-09-29T22:12:24.086 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU= 2023-09-29T22:12:24.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU=', 'Content-Length': '0'} 2023-09-29T22:12:24.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39da945f9347540-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.089 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.089 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:24.089 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:24.090 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:24.090 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:24.090 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.090 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.090 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.090 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.090 DEBUG:boto:Method: GET 2023-09-29T22:12:24.090 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.090 DEBUG:boto:Data: 2023-09-29T22:12:24.091 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.091 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.091 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.091 DEBUG:boto:Params: {} 2023-09-29T22:12:24.091 DEBUG:boto:Token: None 2023-09-29T22:12:24.091 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/?versions 2023-09-29T22:12:24.091 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU= 2023-09-29T22:12:24.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU=', 'Content-Length': '0'} 2023-09-29T22:12:24.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803a53c8e9be2086-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.094 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.094 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.094 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.095 DEBUG:boto:path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.095 DEBUG:boto:auth_path=/dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.095 DEBUG:boto:Method: GET 2023-09-29T22:12:24.095 DEBUG:boto:Path: /dmuhsu-10/?versions&max-keys=999 2023-09-29T22:12:24.095 DEBUG:boto:Data: 2023-09-29T22:12:24.095 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.095 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.095 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.095 DEBUG:boto:Params: {} 2023-09-29T22:12:24.095 DEBUG:boto:Token: None 2023-09-29T22:12:24.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/?versions 2023-09-29T22:12:24.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU= 2023-09-29T22:12:24.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GL1flhND1/VxO/sQQNL3WrO9ScU=', 'Content-Length': '0'} 2023-09-29T22:12:24.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bcf5f5fa7d59a6-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.098 DEBUG:boto:b'dmuhsu-10999false&nulltrue2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.098 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.099 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.099 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.099 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.099 DEBUG:boto:Method: GET 2023-09-29T22:12:24.099 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.099 DEBUG:boto:Data: 2023-09-29T22:12:24.099 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.099 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.099 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.099 DEBUG:boto:Params: {} 2023-09-29T22:12:24.099 DEBUG:boto:Token: None 2023-09-29T22:12:24.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.100 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI= 2023-09-29T22:12:24.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI=', 'Content-Length': '0'} 2023-09-29T22:12:24.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79c8c8daf311b2b-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.146 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.147 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.147 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.147 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.147 DEBUG:boto:Method: GET 2023-09-29T22:12:24.147 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.147 DEBUG:boto:Data: 2023-09-29T22:12:24.147 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.147 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.147 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.147 DEBUG:boto:Params: {} 2023-09-29T22:12:24.148 DEBUG:boto:Token: None 2023-09-29T22:12:24.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI= 2023-09-29T22:12:24.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI=', 'Content-Length': '0'} 2023-09-29T22:12:24.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002db6344ef3703e05-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.150 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.150 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.151 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.151 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.151 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.151 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.151 DEBUG:boto:Data: 2023-09-29T22:12:24.151 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.151 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.151 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.151 DEBUG:boto:Params: {} 2023-09-29T22:12:24.151 DEBUG:boto:Token: None 2023-09-29T22:12:24.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.152 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6VFrP9HLB3EQ072gL1wRW8tXM3Q= 2023-09-29T22:12:24.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6VFrP9HLB3EQ072gL1wRW8tXM3Q=', 'Content-Length': '0'} 2023-09-29T22:12:24.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6836dfc63497d6c-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.154 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.154 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.154 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.154 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.154 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.154 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.154 DEBUG:boto:Data: 2023-09-29T22:12:24.154 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.155 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.155 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.155 DEBUG:boto:Params: {} 2023-09-29T22:12:24.155 DEBUG:boto:Token: None 2023-09-29T22:12:24.155 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6VFrP9HLB3EQ072gL1wRW8tXM3Q= 2023-09-29T22:12:24.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6VFrP9HLB3EQ072gL1wRW8tXM3Q=', 'Content-Length': '0'} 2023-09-29T22:12:24.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dca830b8598ff1a8-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.157 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.157 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.157 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.158 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.158 DEBUG:boto:Method: GET 2023-09-29T22:12:24.158 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.158 DEBUG:boto:Data: 2023-09-29T22:12:24.158 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.158 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.158 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.158 DEBUG:boto:Params: {} 2023-09-29T22:12:24.158 DEBUG:boto:Token: None 2023-09-29T22:12:24.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.159 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI= 2023-09-29T22:12:24.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI=', 'Content-Length': '0'} 2023-09-29T22:12:24.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006dc5614bbbda81c-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.160 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.161 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.161 DEBUG:boto:path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.161 DEBUG:boto:auth_path=/dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.161 DEBUG:boto:Method: GET 2023-09-29T22:12:24.161 DEBUG:boto:Path: /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.161 DEBUG:boto:Data: 2023-09-29T22:12:24.161 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.161 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.161 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.161 DEBUG:boto:Params: {} 2023-09-29T22:12:24.162 DEBUG:boto:Token: None 2023-09-29T22:12:24.162 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26?versionId=null 2023-09-29T22:12:24.162 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI= 2023-09-29T22:12:24.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D/00Snxrb1+KK2mGtRh4/Ux2nsI=', 'Content-Length': '0'} 2023-09-29T22:12:24.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec0535c8e427a24-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.164 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.164 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.164 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.164 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.164 DEBUG:boto:Method: GET 2023-09-29T22:12:24.164 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.164 DEBUG:boto:Data: 2023-09-29T22:12:24.164 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.165 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.165 DEBUG:boto:Params: {} 2023-09-29T22:12:24.165 DEBUG:boto:Token: None 2023-09-29T22:12:24.165 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe0a5fae3e2cd2b-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.167 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.168 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.168 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.168 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.168 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.168 DEBUG:boto:Method: GET 2023-09-29T22:12:24.168 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.168 DEBUG:boto:Data: 2023-09-29T22:12:24.168 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.168 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.169 DEBUG:boto:Params: {} 2023-09-29T22:12:24.169 DEBUG:boto:Token: None 2023-09-29T22:12:24.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2d38e19d31ee2f-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.171 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.172 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.172 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.172 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.172 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:24.172 DEBUG:boto:Data: 2023-09-29T22:12:24.172 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.173 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.173 DEBUG:boto:Params: {} 2023-09-29T22:12:24.173 DEBUG:boto:Token: None 2023-09-29T22:12:24.173 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26 2023-09-29T22:12:24.173 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s6G+Fs2vWwWS9DODbUJdsX4jq2o= 2023-09-29T22:12:24.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s6G+Fs2vWwWS9DODbUJdsX4jq2o=', 'Content-Length': '0'} 2023-09-29T22:12:24.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000864ac7d295e0cfa3-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.175 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.175 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.175 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.175 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:24.175 DEBUG:boto:Data: 2023-09-29T22:12:24.175 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.176 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.176 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.176 DEBUG:boto:Params: {} 2023-09-29T22:12:24.176 DEBUG:boto:Token: None 2023-09-29T22:12:24.176 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26 2023-09-29T22:12:24.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s6G+Fs2vWwWS9DODbUJdsX4jq2o= 2023-09-29T22:12:24.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s6G+Fs2vWwWS9DODbUJdsX4jq2o=', 'Content-Length': '0'} 2023-09-29T22:12:24.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536e29617d0636db-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.178 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.178 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.178 DEBUG:boto:Method: GET 2023-09-29T22:12:24.178 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:24.179 DEBUG:boto:Data: 2023-09-29T22:12:24.179 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.179 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.179 DEBUG:boto:Params: {} 2023-09-29T22:12:24.179 DEBUG:boto:Token: None 2023-09-29T22:12:24.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26 2023-09-29T22:12:24.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f0YnLn3GMB11gl2cg+E6gCNqcXc= 2023-09-29T22:12:24.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f0YnLn3GMB11gl2cg+E6gCNqcXc=', 'Content-Length': '0'} 2023-09-29T22:12:24.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b3e1237c34560c9-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.222 DEBUG:boto:path=/dmuhsu-10/%26 2023-09-29T22:12:24.223 DEBUG:boto:auth_path=/dmuhsu-10/%26 2023-09-29T22:12:24.223 DEBUG:boto:Method: GET 2023-09-29T22:12:24.223 DEBUG:boto:Path: /dmuhsu-10/%26 2023-09-29T22:12:24.223 DEBUG:boto:Data: 2023-09-29T22:12:24.223 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.223 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.224 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.224 DEBUG:boto:Params: {} 2023-09-29T22:12:24.224 DEBUG:boto:Token: None 2023-09-29T22:12:24.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%26 2023-09-29T22:12:24.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f0YnLn3GMB11gl2cg+E6gCNqcXc= 2023-09-29T22:12:24.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f0YnLn3GMB11gl2cg+E6gCNqcXc=', 'Content-Length': '0'} 2023-09-29T22:12:24.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059f2cc5fbec5969c-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.227 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.228 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.228 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.228 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.228 DEBUG:boto:Method: GET 2023-09-29T22:12:24.228 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.228 DEBUG:boto:Data: 2023-09-29T22:12:24.229 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.229 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.229 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.229 DEBUG:boto:Params: {} 2023-09-29T22:12:24.229 DEBUG:boto:Token: None 2023-09-29T22:12:24.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.230 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac469059e878c90-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.233 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.233 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.233 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.234 DEBUG:boto:path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.234 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.234 DEBUG:boto:Method: GET 2023-09-29T22:12:24.234 DEBUG:boto:Path: /dmuhsu-10/?prefix=%26 2023-09-29T22:12:24.234 DEBUG:boto:Data: 2023-09-29T22:12:24.234 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.235 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.235 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.235 DEBUG:boto:Params: {} 2023-09-29T22:12:24.235 DEBUG:boto:Token: None 2023-09-29T22:12:24.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.235 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d0c562e15aac254-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.238 DEBUG:boto:b'dmuhsu-10&1000false&2023-09-29T22:12:07.760Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:24.239 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.239 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.239 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.239 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.239 DEBUG:boto:Method: GET 2023-09-29T22:12:24.239 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.239 DEBUG:boto:Data: 2023-09-29T22:12:24.239 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.239 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.239 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.240 DEBUG:boto:Params: {} 2023-09-29T22:12:24.240 DEBUG:boto:Token: None 2023-09-29T22:12:24.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A= 2023-09-29T22:12:24.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A=', 'Content-Length': '0'} 2023-09-29T22:12:24.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738c03c78c69ecee-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.282 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.282 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.283 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.283 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.283 DEBUG:boto:Method: GET 2023-09-29T22:12:24.283 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.283 DEBUG:boto:Data: 2023-09-29T22:12:24.283 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.283 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.283 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.283 DEBUG:boto:Params: {} 2023-09-29T22:12:24.283 DEBUG:boto:Token: None 2023-09-29T22:12:24.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A= 2023-09-29T22:12:24.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A=', 'Content-Length': '0'} 2023-09-29T22:12:24.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007772d0f94821d763-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.286 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.286 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.286 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.286 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.286 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.287 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.287 DEBUG:boto:Data: 2023-09-29T22:12:24.287 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.287 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.287 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.287 DEBUG:boto:Params: {} 2023-09-29T22:12:24.287 DEBUG:boto:Token: None 2023-09-29T22:12:24.287 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fV5JOOtAhKLFItFf0fh+j31mdJE= 2023-09-29T22:12:24.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fV5JOOtAhKLFItFf0fh+j31mdJE=', 'Content-Length': '0'} 2023-09-29T22:12:24.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091cb72e7409e0e82-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.290 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.290 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.290 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.290 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.290 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.290 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.290 DEBUG:boto:Data: 2023-09-29T22:12:24.291 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.291 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.291 DEBUG:boto:Params: {} 2023-09-29T22:12:24.291 DEBUG:boto:Token: None 2023-09-29T22:12:24.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.291 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fV5JOOtAhKLFItFf0fh+j31mdJE= 2023-09-29T22:12:24.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fV5JOOtAhKLFItFf0fh+j31mdJE=', 'Content-Length': '0'} 2023-09-29T22:12:24.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000172f62fdf35af0da-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:24.294 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.294 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.294 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.294 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.294 DEBUG:boto:Method: GET 2023-09-29T22:12:24.295 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.295 DEBUG:boto:Data: 2023-09-29T22:12:24.295 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.295 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.295 DEBUG:boto:Params: {} 2023-09-29T22:12:24.296 DEBUG:boto:Token: None 2023-09-29T22:12:24.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A= 2023-09-29T22:12:24.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A=', 'Content-Length': '0'} 2023-09-29T22:12:24.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b29b95a33e9412c-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.298 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.298 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.298 DEBUG:boto:path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.298 DEBUG:boto:auth_path=/dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.299 DEBUG:boto:Method: GET 2023-09-29T22:12:24.299 DEBUG:boto:Path: /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.299 DEBUG:boto:Data: 2023-09-29T22:12:24.299 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.299 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.299 DEBUG:boto:Params: {} 2023-09-29T22:12:24.299 DEBUG:boto:Token: None 2023-09-29T22:12:24.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A?versionId=null 2023-09-29T22:12:24.300 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A= 2023-09-29T22:12:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f+5jD3jzgc4AmQl1p2onfvu9K0A=', 'Content-Length': '0'} 2023-09-29T22:12:24.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a9feee3497a37b1-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.302 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.302 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.302 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.302 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.302 DEBUG:boto:Method: GET 2023-09-29T22:12:24.303 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.303 DEBUG:boto:Data: 2023-09-29T22:12:24.303 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.303 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.303 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.303 DEBUG:boto:Params: {} 2023-09-29T22:12:24.303 DEBUG:boto:Token: None 2023-09-29T22:12:24.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b74ed43453ef43-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.306 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.306 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.307 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.307 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.307 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.307 DEBUG:boto:Method: GET 2023-09-29T22:12:24.307 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.307 DEBUG:boto:Data: 2023-09-29T22:12:24.307 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.307 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.308 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.308 DEBUG:boto:Params: {} 2023-09-29T22:12:24.308 DEBUG:boto:Token: None 2023-09-29T22:12:24.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004138e118f5a1e413-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.310 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.311 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.311 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.311 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.311 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:24.311 DEBUG:boto:Data: 2023-09-29T22:12:24.311 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.312 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.312 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.312 DEBUG:boto:Params: {} 2023-09-29T22:12:24.312 DEBUG:boto:Token: None 2023-09-29T22:12:24.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A 2023-09-29T22:12:24.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fl3TCHsP1XFYqXbMeYuodxDOdAY= 2023-09-29T22:12:24.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fl3TCHsP1XFYqXbMeYuodxDOdAY=', 'Content-Length': '0'} 2023-09-29T22:12:24.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a910b8ba6d5f9004-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.314 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.314 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.315 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.315 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:24.315 DEBUG:boto:Data: 2023-09-29T22:12:24.315 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.315 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.315 DEBUG:boto:Params: {} 2023-09-29T22:12:24.315 DEBUG:boto:Token: None 2023-09-29T22:12:24.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A 2023-09-29T22:12:24.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fl3TCHsP1XFYqXbMeYuodxDOdAY= 2023-09-29T22:12:24.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fl3TCHsP1XFYqXbMeYuodxDOdAY=', 'Content-Length': '0'} 2023-09-29T22:12:24.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d62270a985c0519-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:24.318 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.318 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.318 DEBUG:boto:Method: GET 2023-09-29T22:12:24.318 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:24.318 DEBUG:boto:Data: 2023-09-29T22:12:24.318 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.318 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.319 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.319 DEBUG:boto:Params: {} 2023-09-29T22:12:24.319 DEBUG:boto:Token: None 2023-09-29T22:12:24.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A 2023-09-29T22:12:24.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pED0w9MHKRroF9x6flfA8E99MA4= 2023-09-29T22:12:24.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pED0w9MHKRroF9x6flfA8E99MA4=', 'Content-Length': '0'} 2023-09-29T22:12:24.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e88f0e0c6db21c9-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.321 DEBUG:boto:path=/dmuhsu-10/%3A 2023-09-29T22:12:24.321 DEBUG:boto:auth_path=/dmuhsu-10/%3A 2023-09-29T22:12:24.321 DEBUG:boto:Method: GET 2023-09-29T22:12:24.321 DEBUG:boto:Path: /dmuhsu-10/%3A 2023-09-29T22:12:24.321 DEBUG:boto:Data: 2023-09-29T22:12:24.322 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.322 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.322 DEBUG:boto:Params: {} 2023-09-29T22:12:24.322 DEBUG:boto:Token: None 2023-09-29T22:12:24.322 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/%3A 2023-09-29T22:12:24.322 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pED0w9MHKRroF9x6flfA8E99MA4= 2023-09-29T22:12:24.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pED0w9MHKRroF9x6flfA8E99MA4=', 'Content-Length': '0'} 2023-09-29T22:12:24.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc4bf9f0cee06797-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.366 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.366 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.366 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.367 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.367 DEBUG:boto:Method: GET 2023-09-29T22:12:24.367 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.367 DEBUG:boto:Data: 2023-09-29T22:12:24.367 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.367 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.367 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.367 DEBUG:boto:Params: {} 2023-09-29T22:12:24.368 DEBUG:boto:Token: None 2023-09-29T22:12:24.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a9e2efd03e2d63-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.370 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.371 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.371 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.371 DEBUG:boto:path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.371 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.371 DEBUG:boto:Method: GET 2023-09-29T22:12:24.371 DEBUG:boto:Path: /dmuhsu-10/?prefix=%3A 2023-09-29T22:12:24.371 DEBUG:boto:Data: 2023-09-29T22:12:24.372 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.372 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.372 DEBUG:boto:Params: {} 2023-09-29T22:12:24.372 DEBUG:boto:Token: None 2023-09-29T22:12:24.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9442bd3502a44b-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.375 DEBUG:boto:b'dmuhsu-10:1000false:2023-09-29T22:12:07.709Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:24.375 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.375 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.375 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.376 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.376 DEBUG:boto:Method: GET 2023-09-29T22:12:24.376 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.376 DEBUG:boto:Data: 2023-09-29T22:12:24.376 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.376 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.376 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.376 DEBUG:boto:Params: {} 2023-09-29T22:12:24.377 DEBUG:boto:Token: None 2023-09-29T22:12:24.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.377 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw= 2023-09-29T22:12:24.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw=', 'Content-Length': '0'} 2023-09-29T22:12:24.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebb687c90a42df04-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.422 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.422 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.423 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.423 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.423 DEBUG:boto:Method: GET 2023-09-29T22:12:24.423 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.423 DEBUG:boto:Data: 2023-09-29T22:12:24.423 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.423 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.423 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.423 DEBUG:boto:Params: {} 2023-09-29T22:12:24.424 DEBUG:boto:Token: None 2023-09-29T22:12:24.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw= 2023-09-29T22:12:24.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw=', 'Content-Length': '0'} 2023-09-29T22:12:24.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ea5b8c10973820e-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.426 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.427 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.427 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.427 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.427 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.427 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.427 DEBUG:boto:Data: 2023-09-29T22:12:24.427 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.427 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.427 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.428 DEBUG:boto:Params: {} 2023-09-29T22:12:24.428 DEBUG:boto:Token: None 2023-09-29T22:12:24.428 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sxeQXjfsb3ecuE4EGItVv34vgk4= 2023-09-29T22:12:24.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sxeQXjfsb3ecuE4EGItVv34vgk4=', 'Content-Length': '0'} 2023-09-29T22:12:24.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099ea68b4f1594cbd-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.430 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.430 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.430 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.431 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.431 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.431 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.431 DEBUG:boto:Data: 2023-09-29T22:12:24.431 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.431 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.431 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.431 DEBUG:boto:Params: {} 2023-09-29T22:12:24.432 DEBUG:boto:Token: None 2023-09-29T22:12:24.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.432 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sxeQXjfsb3ecuE4EGItVv34vgk4= 2023-09-29T22:12:24.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sxeQXjfsb3ecuE4EGItVv34vgk4=', 'Content-Length': '0'} 2023-09-29T22:12:24.433 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f136da76cba9d4f5-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:24.434 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.434 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.434 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.434 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.435 DEBUG:boto:Method: GET 2023-09-29T22:12:24.435 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.435 DEBUG:boto:Data: 2023-09-29T22:12:24.435 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.435 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.435 DEBUG:boto:Params: {} 2023-09-29T22:12:24.436 DEBUG:boto:Token: None 2023-09-29T22:12:24.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.436 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw= 2023-09-29T22:12:24.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw=', 'Content-Length': '0'} 2023-09-29T22:12:24.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011f8ac66721a9d36-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.438 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.438 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.438 DEBUG:boto:path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.439 DEBUG:boto:auth_path=/dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.439 DEBUG:boto:Method: GET 2023-09-29T22:12:24.439 DEBUG:boto:Path: /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.439 DEBUG:boto:Data: 2023-09-29T22:12:24.439 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.439 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.439 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.440 DEBUG:boto:Params: {} 2023-09-29T22:12:24.440 DEBUG:boto:Token: None 2023-09-29T22:12:24.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj?versionId=null 2023-09-29T22:12:24.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw= 2023-09-29T22:12:24.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHKocPey7ieuzL6DNUNHCACpePw=', 'Content-Length': '0'} 2023-09-29T22:12:24.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004511cbe4127be959-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.443 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.443 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.443 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.443 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.443 DEBUG:boto:Method: GET 2023-09-29T22:12:24.444 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.444 DEBUG:boto:Data: 2023-09-29T22:12:24.444 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.444 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.444 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.444 DEBUG:boto:Params: {} 2023-09-29T22:12:24.445 DEBUG:boto:Token: None 2023-09-29T22:12:24.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.445 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91880226aaa72b4-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.448 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.448 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.449 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.449 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.449 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.449 DEBUG:boto:Method: GET 2023-09-29T22:12:24.449 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.449 DEBUG:boto:Data: 2023-09-29T22:12:24.450 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.450 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.450 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.450 DEBUG:boto:Params: {} 2023-09-29T22:12:24.450 DEBUG:boto:Token: None 2023-09-29T22:12:24.450 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.451 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005623ad3427f34da1-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.453 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.453 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.453 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.454 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.454 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:24.454 DEBUG:boto:Data: 2023-09-29T22:12:24.454 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.454 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.454 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.454 DEBUG:boto:Params: {} 2023-09-29T22:12:24.454 DEBUG:boto:Token: None 2023-09-29T22:12:24.455 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:24.455 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:C/CSad55y4ATSRHlDFQ79PqfsrY= 2023-09-29T22:12:24.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:C/CSad55y4ATSRHlDFQ79PqfsrY=', 'Content-Length': '0'} 2023-09-29T22:12:24.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a330f8bf3b7e176-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.457 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.457 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.457 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.457 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:24.457 DEBUG:boto:Data: 2023-09-29T22:12:24.457 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.457 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.457 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.458 DEBUG:boto:Params: {} 2023-09-29T22:12:24.458 DEBUG:boto:Token: None 2023-09-29T22:12:24.458 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:24.458 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:C/CSad55y4ATSRHlDFQ79PqfsrY= 2023-09-29T22:12:24.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:C/CSad55y4ATSRHlDFQ79PqfsrY=', 'Content-Length': '0'} 2023-09-29T22:12:24.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c178c59eba2678b-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:24.460 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.460 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.460 DEBUG:boto:Method: GET 2023-09-29T22:12:24.460 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:24.460 DEBUG:boto:Data: 2023-09-29T22:12:24.460 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.460 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.461 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.461 DEBUG:boto:Params: {} 2023-09-29T22:12:24.461 DEBUG:boto:Token: None 2023-09-29T22:12:24.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:24.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NJutaEUVuuF7wty8QB1YxZDIil8= 2023-09-29T22:12:24.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NJutaEUVuuF7wty8QB1YxZDIil8=', 'Content-Length': '0'} 2023-09-29T22:12:24.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea9c355dc07a084f-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.463 DEBUG:boto:path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.463 DEBUG:boto:auth_path=/dmuhsu-10/_myobj 2023-09-29T22:12:24.463 DEBUG:boto:Method: GET 2023-09-29T22:12:24.463 DEBUG:boto:Path: /dmuhsu-10/_myobj 2023-09-29T22:12:24.463 DEBUG:boto:Data: 2023-09-29T22:12:24.463 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.463 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.464 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.464 DEBUG:boto:Params: {} 2023-09-29T22:12:24.464 DEBUG:boto:Token: None 2023-09-29T22:12:24.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/_myobj 2023-09-29T22:12:24.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NJutaEUVuuF7wty8QB1YxZDIil8= 2023-09-29T22:12:24.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NJutaEUVuuF7wty8QB1YxZDIil8=', 'Content-Length': '0'} 2023-09-29T22:12:24.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031442d1ba0cad247-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.466 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.466 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.466 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.466 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.466 DEBUG:boto:Method: GET 2023-09-29T22:12:24.466 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.466 DEBUG:boto:Data: 2023-09-29T22:12:24.467 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.467 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.467 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.467 DEBUG:boto:Params: {} 2023-09-29T22:12:24.467 DEBUG:boto:Token: None 2023-09-29T22:12:24.467 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.467 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e185eb393817e2f0-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.510 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.511 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.511 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.511 DEBUG:boto:path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.511 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.511 DEBUG:boto:Method: GET 2023-09-29T22:12:24.511 DEBUG:boto:Path: /dmuhsu-10/?prefix=_myobj 2023-09-29T22:12:24.511 DEBUG:boto:Data: 2023-09-29T22:12:24.512 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.512 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.512 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.512 DEBUG:boto:Params: {} 2023-09-29T22:12:24.512 DEBUG:boto:Token: None 2023-09-29T22:12:24.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026070bcd99143c34-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.515 DEBUG:boto:b'dmuhsu-10_myobj1000false_myobj2023-09-29T22:12:07.696Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:24.515 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.515 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.515 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.516 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.516 DEBUG:boto:Method: GET 2023-09-29T22:12:24.516 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.516 DEBUG:boto:Data: 2023-09-29T22:12:24.516 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.516 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.516 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.516 DEBUG:boto:Params: {} 2023-09-29T22:12:24.516 DEBUG:boto:Token: None 2023-09-29T22:12:24.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0= 2023-09-29T22:12:24.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0=', 'Content-Length': '0'} 2023-09-29T22:12:24.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c2934869a01314-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.519 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.519 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.519 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.519 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.519 DEBUG:boto:Method: GET 2023-09-29T22:12:24.519 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.519 DEBUG:boto:Data: 2023-09-29T22:12:24.519 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.520 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.520 DEBUG:boto:Params: {} 2023-09-29T22:12:24.520 DEBUG:boto:Token: None 2023-09-29T22:12:24.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.520 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0= 2023-09-29T22:12:24.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0=', 'Content-Length': '0'} 2023-09-29T22:12:24.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002af3f59cc134c417-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.562 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.562 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.563 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.563 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.563 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.563 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.563 DEBUG:boto:Data: 2023-09-29T22:12:24.563 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.563 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.564 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.564 DEBUG:boto:Params: {} 2023-09-29T22:12:24.564 DEBUG:boto:Token: None 2023-09-29T22:12:24.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hYKFZCKi+i87/5YGBP+Fj5W35/Q= 2023-09-29T22:12:24.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hYKFZCKi+i87/5YGBP+Fj5W35/Q=', 'Content-Length': '0'} 2023-09-29T22:12:24.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008944723ab43f4cc0-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.567 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.567 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.567 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.567 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.567 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.567 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.568 DEBUG:boto:Data: 2023-09-29T22:12:24.568 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.568 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.568 DEBUG:boto:Params: {} 2023-09-29T22:12:24.568 DEBUG:boto:Token: None 2023-09-29T22:12:24.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hYKFZCKi+i87/5YGBP+Fj5W35/Q= 2023-09-29T22:12:24.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hYKFZCKi+i87/5YGBP+Fj5W35/Q=', 'Content-Length': '0'} 2023-09-29T22:12:24.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c465677aa95cbef-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:24.571 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.571 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.571 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.571 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.571 DEBUG:boto:Method: GET 2023-09-29T22:12:24.572 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.572 DEBUG:boto:Data: 2023-09-29T22:12:24.572 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.572 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.572 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.572 DEBUG:boto:Params: {} 2023-09-29T22:12:24.573 DEBUG:boto:Token: None 2023-09-29T22:12:24.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.573 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0= 2023-09-29T22:12:24.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0=', 'Content-Length': '0'} 2023-09-29T22:12:24.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000108c3d748bfcb3cc-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.575 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.575 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.575 DEBUG:boto:path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.575 DEBUG:boto:auth_path=/dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.576 DEBUG:boto:Method: GET 2023-09-29T22:12:24.576 DEBUG:boto:Path: /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.576 DEBUG:boto:Data: 2023-09-29T22:12:24.576 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.576 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.576 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.576 DEBUG:boto:Params: {} 2023-09-29T22:12:24.577 DEBUG:boto:Token: None 2023-09-29T22:12:24.577 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj?versionId=null 2023-09-29T22:12:24.577 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0= 2023-09-29T22:12:24.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+EZUBlR+/bprSL7CweCEdU3FRO0=', 'Content-Length': '0'} 2023-09-29T22:12:24.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053d68754034476df-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.579 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.579 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.579 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.579 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.580 DEBUG:boto:Method: GET 2023-09-29T22:12:24.580 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.580 DEBUG:boto:Data: 2023-09-29T22:12:24.580 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.580 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.580 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.581 DEBUG:boto:Params: {} 2023-09-29T22:12:24.581 DEBUG:boto:Token: None 2023-09-29T22:12:24.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.581 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e54628c52d2b276-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.626 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.627 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.627 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.627 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.627 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.628 DEBUG:boto:Method: GET 2023-09-29T22:12:24.628 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.628 DEBUG:boto:Data: 2023-09-29T22:12:24.628 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.628 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.628 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.628 DEBUG:boto:Params: {} 2023-09-29T22:12:24.629 DEBUG:boto:Token: None 2023-09-29T22:12:24.629 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.629 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05919c2d1f3a191-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.632 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.632 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.632 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.632 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.632 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:24.633 DEBUG:boto:Data: 2023-09-29T22:12:24.633 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.633 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.633 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.633 DEBUG:boto:Params: {} 2023-09-29T22:12:24.633 DEBUG:boto:Token: None 2023-09-29T22:12:24.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj 2023-09-29T22:12:24.634 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6CADO+fOvE5MQJBLcVayL2FytG4= 2023-09-29T22:12:24.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6CADO+fOvE5MQJBLcVayL2FytG4=', 'Content-Length': '0'} 2023-09-29T22:12:24.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6531ad79ac4c096-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.636 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.636 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.636 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.636 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:24.636 DEBUG:boto:Data: 2023-09-29T22:12:24.636 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.637 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.637 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.637 DEBUG:boto:Params: {} 2023-09-29T22:12:24.637 DEBUG:boto:Token: None 2023-09-29T22:12:24.637 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj 2023-09-29T22:12:24.637 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6CADO+fOvE5MQJBLcVayL2FytG4= 2023-09-29T22:12:24.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6CADO+fOvE5MQJBLcVayL2FytG4=', 'Content-Length': '0'} 2023-09-29T22:12:24.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216e1d1a02b95d1a-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:24.640 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.640 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.640 DEBUG:boto:Method: GET 2023-09-29T22:12:24.640 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:24.640 DEBUG:boto:Data: 2023-09-29T22:12:24.640 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.640 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.640 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.641 DEBUG:boto:Params: {} 2023-09-29T22:12:24.641 DEBUG:boto:Token: None 2023-09-29T22:12:24.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj 2023-09-29T22:12:24.641 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LdmcC0X9RYoMaBo0/GCzlMf5oqo= 2023-09-29T22:12:24.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LdmcC0X9RYoMaBo0/GCzlMf5oqo=', 'Content-Length': '0'} 2023-09-29T22:12:24.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065b8fc5390995b38-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.643 DEBUG:boto:path=/dmuhsu-10/myobj 2023-09-29T22:12:24.643 DEBUG:boto:auth_path=/dmuhsu-10/myobj 2023-09-29T22:12:24.643 DEBUG:boto:Method: GET 2023-09-29T22:12:24.644 DEBUG:boto:Path: /dmuhsu-10/myobj 2023-09-29T22:12:24.644 DEBUG:boto:Data: 2023-09-29T22:12:24.644 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.644 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.644 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.644 DEBUG:boto:Params: {} 2023-09-29T22:12:24.644 DEBUG:boto:Token: None 2023-09-29T22:12:24.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/myobj 2023-09-29T22:12:24.645 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LdmcC0X9RYoMaBo0/GCzlMf5oqo= 2023-09-29T22:12:24.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LdmcC0X9RYoMaBo0/GCzlMf5oqo=', 'Content-Length': '0'} 2023-09-29T22:12:24.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b602f81fa1d69f7d-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.690 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.690 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.691 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.691 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.691 DEBUG:boto:Method: GET 2023-09-29T22:12:24.691 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.691 DEBUG:boto:Data: 2023-09-29T22:12:24.691 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.691 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:24.692 DEBUG:boto:Port: 8001 2023-09-29T22:12:24.692 DEBUG:boto:Params: {} 2023-09-29T22:12:24.692 DEBUG:boto:Token: None 2023-09-29T22:12:24.692 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.692 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b56ea2740f1f18d-0065174bc8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.695 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.696 DEBUG:boto:path=/dmuhsu-10/ 2023-09-29T22:12:24.696 DEBUG:boto:auth_path=/dmuhsu-10/ 2023-09-29T22:12:24.697 DEBUG:boto:path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.697 DEBUG:boto:auth_path=/dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.697 DEBUG:boto:Method: GET 2023-09-29T22:12:24.697 DEBUG:boto:Path: /dmuhsu-10/?prefix=myobj 2023-09-29T22:12:24.697 DEBUG:boto:Data: 2023-09-29T22:12:24.697 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.698 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.698 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.698 DEBUG:boto:Params: {} 2023-09-29T22:12:24.698 DEBUG:boto:Token: None 2023-09-29T22:12:24.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-10/ 2023-09-29T22:12:24.699 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY= 2023-09-29T22:12:24.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I7reZ5+Jc0y2Qhnj/eVrSBcaZKY=', 'Content-Length': '0'} 2023-09-29T22:12:24.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0001c62eeb2f5a-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.702 DEBUG:boto:b'dmuhsu-10myobj1000falsemyobj2023-09-29T22:12:07.682Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.702 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-10 zones={test-zone3, test-zone1} 2023-09-29T22:12:24.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:24.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:24.703 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:24.759 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:24.759 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:24.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:24.794+0000 7eff82136880 1 bucket sync caught up with source: 2023-09-29T22:12:24.794 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.41.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.15.6, , ] 2023-09-29T22:12:24.794 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.41.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.15.6, , ] 2023-09-29T22:12:24.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:24.794+0000 7eff82136880 0 bucket checkpoint complete 2023-09-29T22:12:24.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-11 zones={test-zone1, test-zone2} 2023-09-29T22:12:24.806 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.806 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.807 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.807 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:24.807 DEBUG:boto:Data: 2023-09-29T22:12:24.807 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.807 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.808 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.808 DEBUG:boto:Params: {} 2023-09-29T22:12:24.808 DEBUG:boto:Token: None 2023-09-29T22:12:24.808 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DmBpo7/KCUo9hSQGUHRM6uCcf9g= 2023-09-29T22:12:24.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DmBpo7/KCUo9hSQGUHRM6uCcf9g=', 'Content-Length': '0'} 2023-09-29T22:12:24.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c176af16a7ea76ef-0065174bc8-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.812 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.812 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.812 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.812 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:24.812 DEBUG:boto:Data: 2023-09-29T22:12:24.813 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.813 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.813 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.813 DEBUG:boto:Params: {} 2023-09-29T22:12:24.813 DEBUG:boto:Token: None 2023-09-29T22:12:24.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.814 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DmBpo7/KCUo9hSQGUHRM6uCcf9g= 2023-09-29T22:12:24.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DmBpo7/KCUo9hSQGUHRM6uCcf9g=', 'Content-Length': '0'} 2023-09-29T22:12:24.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d044b2d9db64a722-0065174bc8-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:24.817 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.817 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.817 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.817 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.817 DEBUG:boto:Method: GET 2023-09-29T22:12:24.817 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.817 DEBUG:boto:Data: 2023-09-29T22:12:24.818 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.818 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.818 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.818 DEBUG:boto:Params: {} 2023-09-29T22:12:24.818 DEBUG:boto:Token: None 2023-09-29T22:12:24.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/?versions 2023-09-29T22:12:24.819 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y= 2023-09-29T22:12:24.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y=', 'Content-Length': '0'} 2023-09-29T22:12:24.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0812aea1e4fee78-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.821 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:24.822 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:24.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:24.823 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.823 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.823 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.823 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.823 DEBUG:boto:Method: GET 2023-09-29T22:12:24.823 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.824 DEBUG:boto:Data: 2023-09-29T22:12:24.824 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.824 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.824 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.824 DEBUG:boto:Params: {} 2023-09-29T22:12:24.824 DEBUG:boto:Token: None 2023-09-29T22:12:24.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/?versions 2023-09-29T22:12:24.825 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y= 2023-09-29T22:12:24.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y=', 'Content-Length': '0'} 2023-09-29T22:12:24.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001073f781992d2275-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.827 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:24.828 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.829 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.829 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.829 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.829 DEBUG:boto:Method: GET 2023-09-29T22:12:24.829 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.829 DEBUG:boto:Data: 2023-09-29T22:12:24.829 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.830 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.830 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.830 DEBUG:boto:Params: {} 2023-09-29T22:12:24.830 DEBUG:boto:Token: None 2023-09-29T22:12:24.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/?versions 2023-09-29T22:12:24.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y= 2023-09-29T22:12:24.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y=', 'Content-Length': '0'} 2023-09-29T22:12:24.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc5910e3457c079-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.833 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.834 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.834 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.834 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.834 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.834 DEBUG:boto:Method: GET 2023-09-29T22:12:24.835 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:24.835 DEBUG:boto:Data: 2023-09-29T22:12:24.835 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.835 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.835 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.835 DEBUG:boto:Params: {} 2023-09-29T22:12:24.836 DEBUG:boto:Token: None 2023-09-29T22:12:24.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/?versions 2023-09-29T22:12:24.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y= 2023-09-29T22:12:24.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O82lOsNF26/ri0UUvVfjitFuF1Y=', 'Content-Length': '0'} 2023-09-29T22:12:24.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bffdc7185c42bad-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.839 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.839 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.840 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.840 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.840 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.840 DEBUG:boto:Method: GET 2023-09-29T22:12:24.840 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.840 DEBUG:boto:Data: 2023-09-29T22:12:24.840 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.841 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.841 DEBUG:boto:Params: {} 2023-09-29T22:12:24.841 DEBUG:boto:Token: None 2023-09-29T22:12:24.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.841 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM= 2023-09-29T22:12:24.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM=', 'Content-Length': '0'} 2023-09-29T22:12:24.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5781e478c3ab235-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.844 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.844 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.844 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.844 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.844 DEBUG:boto:Method: GET 2023-09-29T22:12:24.844 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.844 DEBUG:boto:Data: 2023-09-29T22:12:24.844 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.845 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.845 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.845 DEBUG:boto:Params: {} 2023-09-29T22:12:24.845 DEBUG:boto:Token: None 2023-09-29T22:12:24.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM= 2023-09-29T22:12:24.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM=', 'Content-Length': '0'} 2023-09-29T22:12:24.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000804c7dd73a83b44a-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.894 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.895 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.895 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.895 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.895 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.895 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.895 DEBUG:boto:Data: 2023-09-29T22:12:24.896 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.896 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.896 DEBUG:boto:Params: {} 2023-09-29T22:12:24.896 DEBUG:boto:Token: None 2023-09-29T22:12:24.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hSpqSLYGBu/W86ApUVbk8EyzA5k= 2023-09-29T22:12:24.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hSpqSLYGBu/W86ApUVbk8EyzA5k=', 'Content-Length': '0'} 2023-09-29T22:12:24.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000787cf6dfc675c095-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.899 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.899 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.899 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.899 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.899 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.899 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.899 DEBUG:boto:Data: 2023-09-29T22:12:24.900 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.900 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.900 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.900 DEBUG:boto:Params: {} 2023-09-29T22:12:24.900 DEBUG:boto:Token: None 2023-09-29T22:12:24.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.901 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hSpqSLYGBu/W86ApUVbk8EyzA5k= 2023-09-29T22:12:24.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hSpqSLYGBu/W86ApUVbk8EyzA5k=', 'Content-Length': '0'} 2023-09-29T22:12:24.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094531ce8e7d3fa09-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.903 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.903 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.903 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.903 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.903 DEBUG:boto:Method: GET 2023-09-29T22:12:24.903 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.903 DEBUG:boto:Data: 2023-09-29T22:12:24.904 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.904 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.904 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.904 DEBUG:boto:Params: {} 2023-09-29T22:12:24.904 DEBUG:boto:Token: None 2023-09-29T22:12:24.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM= 2023-09-29T22:12:24.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM=', 'Content-Length': '0'} 2023-09-29T22:12:24.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073199ab3df929256-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.946 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.947 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.947 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.947 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.947 DEBUG:boto:Method: GET 2023-09-29T22:12:24.947 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.947 DEBUG:boto:Data: 2023-09-29T22:12:24.947 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.947 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.947 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.947 DEBUG:boto:Params: {} 2023-09-29T22:12:24.948 DEBUG:boto:Token: None 2023-09-29T22:12:24.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:24.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM= 2023-09-29T22:12:24.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5oHAvjo/RUx8oGNbcm8J7BrmCIM=', 'Content-Length': '0'} 2023-09-29T22:12:24.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a31f6773b906b665-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.950 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.950 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.951 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.951 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.951 DEBUG:boto:Method: GET 2023-09-29T22:12:24.951 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.951 DEBUG:boto:Data: 2023-09-29T22:12:24.951 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.951 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.951 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.952 DEBUG:boto:Params: {} 2023-09-29T22:12:24.952 DEBUG:boto:Token: None 2023-09-29T22:12:24.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw= 2023-09-29T22:12:24.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw=', 'Content-Length': '0'} 2023-09-29T22:12:24.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003731c652ece088e7-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.954 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.954 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.955 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.955 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.955 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.955 DEBUG:boto:Method: GET 2023-09-29T22:12:24.955 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.955 DEBUG:boto:Data: 2023-09-29T22:12:24.955 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.955 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.955 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.955 DEBUG:boto:Params: {} 2023-09-29T22:12:24.956 DEBUG:boto:Token: None 2023-09-29T22:12:24.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw= 2023-09-29T22:12:24.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw=', 'Content-Length': '0'} 2023-09-29T22:12:24.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f09f67b62029e58f-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.958 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.959 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.959 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.959 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.959 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:24.959 DEBUG:boto:Data: 2023-09-29T22:12:24.959 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.960 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.960 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.960 DEBUG:boto:Params: {} 2023-09-29T22:12:24.960 DEBUG:boto:Token: None 2023-09-29T22:12:24.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26 2023-09-29T22:12:24.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HgVOXP0r3e3X8rWofrkB9mtNd/o= 2023-09-29T22:12:24.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HgVOXP0r3e3X8rWofrkB9mtNd/o=', 'Content-Length': '0'} 2023-09-29T22:12:24.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2d6dff7a6cde10-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.962 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.962 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.962 DEBUG:boto:Method: HEAD 2023-09-29T22:12:24.963 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:24.963 DEBUG:boto:Data: 2023-09-29T22:12:24.963 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.963 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.963 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.963 DEBUG:boto:Params: {} 2023-09-29T22:12:24.963 DEBUG:boto:Token: None 2023-09-29T22:12:24.963 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26 2023-09-29T22:12:24.963 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HgVOXP0r3e3X8rWofrkB9mtNd/o= 2023-09-29T22:12:24.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HgVOXP0r3e3X8rWofrkB9mtNd/o=', 'Content-Length': '0'} 2023-09-29T22:12:24.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7f3dc31562d72a0-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:24.966 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.966 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.966 DEBUG:boto:Method: GET 2023-09-29T22:12:24.966 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:24.966 DEBUG:boto:Data: 2023-09-29T22:12:24.966 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.966 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.966 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.966 DEBUG:boto:Params: {} 2023-09-29T22:12:24.967 DEBUG:boto:Token: None 2023-09-29T22:12:24.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26 2023-09-29T22:12:24.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZxdgFDGtDxalgN3stP+xSFU6Qos= 2023-09-29T22:12:24.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZxdgFDGtDxalgN3stP+xSFU6Qos=', 'Content-Length': '0'} 2023-09-29T22:12:24.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0dc922ade90b1f9-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.969 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:24.969 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:24.969 DEBUG:boto:Method: GET 2023-09-29T22:12:24.969 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:24.969 DEBUG:boto:Data: 2023-09-29T22:12:24.969 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.969 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.969 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.969 DEBUG:boto:Params: {} 2023-09-29T22:12:24.970 DEBUG:boto:Token: None 2023-09-29T22:12:24.970 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%26 2023-09-29T22:12:24.970 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZxdgFDGtDxalgN3stP+xSFU6Qos= 2023-09-29T22:12:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZxdgFDGtDxalgN3stP+xSFU6Qos=', 'Content-Length': '0'} 2023-09-29T22:12:24.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000245345d39cce1231-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.972 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.972 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.972 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.972 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.972 DEBUG:boto:Method: GET 2023-09-29T22:12:24.972 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.972 DEBUG:boto:Data: 2023-09-29T22:12:24.972 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.973 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.973 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.973 DEBUG:boto:Params: {} 2023-09-29T22:12:24.973 DEBUG:boto:Token: None 2023-09-29T22:12:24.973 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw= 2023-09-29T22:12:24.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw=', 'Content-Length': '0'} 2023-09-29T22:12:24.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4aa80def93e23f7-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.975 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.976 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:24.976 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:24.976 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.976 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.976 DEBUG:boto:Method: GET 2023-09-29T22:12:24.976 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:24.976 DEBUG:boto:Data: 2023-09-29T22:12:24.976 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.977 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.977 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.977 DEBUG:boto:Params: {} 2023-09-29T22:12:24.977 DEBUG:boto:Token: None 2023-09-29T22:12:24.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/ 2023-09-29T22:12:24.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw= 2023-09-29T22:12:24.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PkQkbdpa/GLOdSG2zpEXN+T87Sw=', 'Content-Length': '0'} 2023-09-29T22:12:24.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d2e0474dbb7d4e-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.980 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:24.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:24.980 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:24.980 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:24.980 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.981 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.981 DEBUG:boto:Method: GET 2023-09-29T22:12:24.981 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.981 DEBUG:boto:Data: 2023-09-29T22:12:24.981 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:24.981 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.981 DEBUG:boto:Params: {} 2023-09-29T22:12:24.981 DEBUG:boto:Token: None 2023-09-29T22:12:24.982 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.982 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xEOcvCtqxgmMiCj05N+Dsosdk/k= 2023-09-29T22:12:24.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xEOcvCtqxgmMiCj05N+Dsosdk/k=', 'Content-Length': '0'} 2023-09-29T22:12:24.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000641f50c6f764dd77-0065174bc8-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:24.984 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:24.984 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:24.984 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.984 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.984 DEBUG:boto:Method: GET 2023-09-29T22:12:24.984 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.985 DEBUG:boto:Data: 2023-09-29T22:12:24.985 DEBUG:boto:Headers: {} 2023-09-29T22:12:24.985 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:24.985 DEBUG:boto:Port: 8000 2023-09-29T22:12:24.985 DEBUG:boto:Params: {} 2023-09-29T22:12:24.985 DEBUG:boto:Token: None 2023-09-29T22:12:24.985 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:24 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:24.985 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xEOcvCtqxgmMiCj05N+Dsosdk/k= 2023-09-29T22:12:24.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xEOcvCtqxgmMiCj05N+Dsosdk/k=', 'Content-Length': '0'} 2023-09-29T22:12:24.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000549b56d2f2566f5f-0065174bc8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.030 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.031 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.031 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.031 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.031 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.031 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.031 DEBUG:boto:Data: 2023-09-29T22:12:25.031 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.031 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.031 DEBUG:boto:Params: {} 2023-09-29T22:12:25.032 DEBUG:boto:Token: None 2023-09-29T22:12:25.032 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.032 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8= 2023-09-29T22:12:25.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8=', 'Content-Length': '0'} 2023-09-29T22:12:25.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070e08e5509c05f98-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.034 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.034 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.034 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.034 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.035 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.035 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.035 DEBUG:boto:Data: 2023-09-29T22:12:25.035 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.035 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.035 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.035 DEBUG:boto:Params: {} 2023-09-29T22:12:25.035 DEBUG:boto:Token: None 2023-09-29T22:12:25.035 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.035 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8= 2023-09-29T22:12:25.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8=', 'Content-Length': '0'} 2023-09-29T22:12:25.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000996641faab54ec72-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:25.038 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.038 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.038 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.038 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.038 DEBUG:boto:Method: GET 2023-09-29T22:12:25.038 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.038 DEBUG:boto:Data: 2023-09-29T22:12:25.038 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.039 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.039 DEBUG:boto:Params: {} 2023-09-29T22:12:25.039 DEBUG:boto:Token: None 2023-09-29T22:12:25.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.039 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb04b14056130159-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.041 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.041 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.041 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.041 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.041 DEBUG:boto:Method: GET 2023-09-29T22:12:25.042 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.042 DEBUG:boto:Data: 2023-09-29T22:12:25.042 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.042 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.042 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.042 DEBUG:boto:Params: {} 2023-09-29T22:12:25.042 DEBUG:boto:Token: None 2023-09-29T22:12:25.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.042 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002636643de1051cc4-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.045 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.045 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.045 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.045 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.045 DEBUG:boto:Method: GET 2023-09-29T22:12:25.045 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.045 DEBUG:boto:Data: 2023-09-29T22:12:25.045 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.045 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.046 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.046 DEBUG:boto:Params: {} 2023-09-29T22:12:25.046 DEBUG:boto:Token: None 2023-09-29T22:12:25.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.046 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099dc50cab709a50b-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.048 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.049 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.049 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.049 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.049 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.049 DEBUG:boto:Method: GET 2023-09-29T22:12:25.049 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.049 DEBUG:boto:Data: 2023-09-29T22:12:25.050 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.050 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.050 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.050 DEBUG:boto:Params: {} 2023-09-29T22:12:25.050 DEBUG:boto:Token: None 2023-09-29T22:12:25.050 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.050 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fc519392ae3305d-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.053 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.053 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.054 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.054 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.054 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.054 DEBUG:boto:Data: 2023-09-29T22:12:25.054 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.054 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.054 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.054 DEBUG:boto:Params: {} 2023-09-29T22:12:25.055 DEBUG:boto:Token: None 2023-09-29T22:12:25.055 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A= 2023-09-29T22:12:25.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A=', 'Content-Length': '0'} 2023-09-29T22:12:25.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003308c0d43b367ad-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.057 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.057 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.057 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.057 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.057 DEBUG:boto:Data: 2023-09-29T22:12:25.057 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.057 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.057 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.057 DEBUG:boto:Params: {} 2023-09-29T22:12:25.058 DEBUG:boto:Token: None 2023-09-29T22:12:25.058 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A= 2023-09-29T22:12:25.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A=', 'Content-Length': '0'} 2023-09-29T22:12:25.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032d3ad8c02ff85e6-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:25.060 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.060 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.060 DEBUG:boto:Method: GET 2023-09-29T22:12:25.060 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.060 DEBUG:boto:Data: 2023-09-29T22:12:25.060 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.061 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.061 DEBUG:boto:Params: {} 2023-09-29T22:12:25.061 DEBUG:boto:Token: None 2023-09-29T22:12:25.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E= 2023-09-29T22:12:25.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E=', 'Content-Length': '0'} 2023-09-29T22:12:25.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090f6a88c1f8ebcb4-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.063 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.063 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.063 DEBUG:boto:Method: GET 2023-09-29T22:12:25.063 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.063 DEBUG:boto:Data: 2023-09-29T22:12:25.063 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.064 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.064 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.064 DEBUG:boto:Params: {} 2023-09-29T22:12:25.064 DEBUG:boto:Token: None 2023-09-29T22:12:25.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E= 2023-09-29T22:12:25.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E=', 'Content-Length': '0'} 2023-09-29T22:12:25.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007552956b2c56f429-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.066 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.066 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.066 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.066 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.067 DEBUG:boto:Method: GET 2023-09-29T22:12:25.067 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.067 DEBUG:boto:Data: 2023-09-29T22:12:25.067 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.067 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.067 DEBUG:boto:Params: {} 2023-09-29T22:12:25.067 DEBUG:boto:Token: None 2023-09-29T22:12:25.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06a1119a56cb632-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.114 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.115 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.115 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.115 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.115 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.115 DEBUG:boto:Method: GET 2023-09-29T22:12:25.115 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.116 DEBUG:boto:Data: 2023-09-29T22:12:25.116 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.116 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.116 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.116 DEBUG:boto:Params: {} 2023-09-29T22:12:25.116 DEBUG:boto:Token: None 2023-09-29T22:12:25.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.117 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9bd6055590d090c-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.120 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.120 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.120 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.120 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.120 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.121 DEBUG:boto:Method: GET 2023-09-29T22:12:25.121 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.121 DEBUG:boto:Data: 2023-09-29T22:12:25.121 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.121 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.121 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.121 DEBUG:boto:Params: {} 2023-09-29T22:12:25.121 DEBUG:boto:Token: None 2023-09-29T22:12:25.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.122 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000725257422e75bc95-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.123 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.124 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.124 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.124 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.124 DEBUG:boto:Method: GET 2023-09-29T22:12:25.124 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.124 DEBUG:boto:Data: 2023-09-29T22:12:25.124 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.124 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.125 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.125 DEBUG:boto:Params: {} 2023-09-29T22:12:25.125 DEBUG:boto:Token: None 2023-09-29T22:12:25.125 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.125 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060857a2cb41bc19c-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.127 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.127 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.128 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.128 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.128 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.128 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.128 DEBUG:boto:Data: 2023-09-29T22:12:25.128 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.128 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.128 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.128 DEBUG:boto:Params: {} 2023-09-29T22:12:25.129 DEBUG:boto:Token: None 2023-09-29T22:12:25.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.129 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA= 2023-09-29T22:12:25.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA=', 'Content-Length': '0'} 2023-09-29T22:12:25.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c662fa3a014285-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.131 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.131 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.131 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.131 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.131 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.131 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.131 DEBUG:boto:Data: 2023-09-29T22:12:25.132 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.132 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.132 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.132 DEBUG:boto:Params: {} 2023-09-29T22:12:25.132 DEBUG:boto:Token: None 2023-09-29T22:12:25.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.132 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA= 2023-09-29T22:12:25.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA=', 'Content-Length': '0'} 2023-09-29T22:12:25.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daca9629696a1568-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.135 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.135 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.135 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.135 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.135 DEBUG:boto:Method: GET 2023-09-29T22:12:25.135 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.135 DEBUG:boto:Data: 2023-09-29T22:12:25.135 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.135 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.136 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.136 DEBUG:boto:Params: {} 2023-09-29T22:12:25.136 DEBUG:boto:Token: None 2023-09-29T22:12:25.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.136 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd35eaef08d1a97a-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.182 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.183 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.183 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.183 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.183 DEBUG:boto:Method: GET 2023-09-29T22:12:25.183 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.183 DEBUG:boto:Data: 2023-09-29T22:12:25.183 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.184 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.184 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.184 DEBUG:boto:Params: {} 2023-09-29T22:12:25.184 DEBUG:boto:Token: None 2023-09-29T22:12:25.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028a0c83984effa17-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.187 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.187 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.187 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.187 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.187 DEBUG:boto:Method: GET 2023-09-29T22:12:25.187 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.187 DEBUG:boto:Data: 2023-09-29T22:12:25.188 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.188 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.188 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.188 DEBUG:boto:Params: {} 2023-09-29T22:12:25.188 DEBUG:boto:Token: None 2023-09-29T22:12:25.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.188 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006138b2271d8f203d-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.191 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.191 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.191 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.192 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.192 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.192 DEBUG:boto:Method: GET 2023-09-29T22:12:25.192 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.192 DEBUG:boto:Data: 2023-09-29T22:12:25.192 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.192 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.192 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.192 DEBUG:boto:Params: {} 2023-09-29T22:12:25.193 DEBUG:boto:Token: None 2023-09-29T22:12:25.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8eb37d471c8d80-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.196 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.196 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.196 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.197 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.197 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.197 DEBUG:boto:Data: 2023-09-29T22:12:25.197 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.197 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.197 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.197 DEBUG:boto:Params: {} 2023-09-29T22:12:25.197 DEBUG:boto:Token: None 2023-09-29T22:12:25.198 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.198 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU= 2023-09-29T22:12:25.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU=', 'Content-Length': '0'} 2023-09-29T22:12:25.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2fd0c3eba6d9297-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.200 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.200 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.200 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.200 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.200 DEBUG:boto:Data: 2023-09-29T22:12:25.201 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.201 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.201 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.201 DEBUG:boto:Params: {} 2023-09-29T22:12:25.201 DEBUG:boto:Token: None 2023-09-29T22:12:25.201 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU= 2023-09-29T22:12:25.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU=', 'Content-Length': '0'} 2023-09-29T22:12:25.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ca4b810b1ce575b-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.204 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.204 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.204 DEBUG:boto:Method: GET 2023-09-29T22:12:25.204 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.204 DEBUG:boto:Data: 2023-09-29T22:12:25.204 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.205 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.205 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.205 DEBUG:boto:Params: {} 2023-09-29T22:12:25.205 DEBUG:boto:Token: None 2023-09-29T22:12:25.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.205 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI= 2023-09-29T22:12:25.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI=', 'Content-Length': '0'} 2023-09-29T22:12:25.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010fe9a60329812d4-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.207 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.207 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.207 DEBUG:boto:Method: GET 2023-09-29T22:12:25.207 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.208 DEBUG:boto:Data: 2023-09-29T22:12:25.208 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.208 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.208 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.208 DEBUG:boto:Params: {} 2023-09-29T22:12:25.208 DEBUG:boto:Token: None 2023-09-29T22:12:25.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI= 2023-09-29T22:12:25.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI=', 'Content-Length': '0'} 2023-09-29T22:12:25.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080e12f267c3a62b8-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.254 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.255 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.255 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.255 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.255 DEBUG:boto:Method: GET 2023-09-29T22:12:25.255 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.255 DEBUG:boto:Data: 2023-09-29T22:12:25.255 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.255 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.256 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.256 DEBUG:boto:Params: {} 2023-09-29T22:12:25.256 DEBUG:boto:Token: None 2023-09-29T22:12:25.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015630cd05e30bd5b-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.258 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.259 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.259 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.259 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.259 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.259 DEBUG:boto:Method: GET 2023-09-29T22:12:25.260 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.260 DEBUG:boto:Data: 2023-09-29T22:12:25.260 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.260 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.260 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.260 DEBUG:boto:Params: {} 2023-09-29T22:12:25.260 DEBUG:boto:Token: None 2023-09-29T22:12:25.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.261 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005278c9cc262f1428-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.263 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:25.264 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.264 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.264 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.264 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.264 DEBUG:boto:Method: GET 2023-09-29T22:12:25.264 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.265 DEBUG:boto:Data: 2023-09-29T22:12:25.265 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.265 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.265 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.265 DEBUG:boto:Params: {} 2023-09-29T22:12:25.265 DEBUG:boto:Token: None 2023-09-29T22:12:25.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.265 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4331c723e589d38-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.314 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.315 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.315 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.315 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.315 DEBUG:boto:Method: GET 2023-09-29T22:12:25.315 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.315 DEBUG:boto:Data: 2023-09-29T22:12:25.315 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.315 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.316 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.316 DEBUG:boto:Params: {} 2023-09-29T22:12:25.316 DEBUG:boto:Token: None 2023-09-29T22:12:25.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002292d265a01b6899-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.318 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.319 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.319 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.319 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.319 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.319 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.319 DEBUG:boto:Data: 2023-09-29T22:12:25.319 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.319 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.320 DEBUG:boto:Params: {} 2023-09-29T22:12:25.320 DEBUG:boto:Token: None 2023-09-29T22:12:25.320 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo= 2023-09-29T22:12:25.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo=', 'Content-Length': '0'} 2023-09-29T22:12:25.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c949993158adba1e-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.322 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.322 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.322 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.322 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.322 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.322 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.323 DEBUG:boto:Data: 2023-09-29T22:12:25.323 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.323 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.323 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.323 DEBUG:boto:Params: {} 2023-09-29T22:12:25.323 DEBUG:boto:Token: None 2023-09-29T22:12:25.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo= 2023-09-29T22:12:25.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo=', 'Content-Length': '0'} 2023-09-29T22:12:25.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008e0b4680d45a8d1-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:25.325 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.326 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.326 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.326 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.326 DEBUG:boto:Method: GET 2023-09-29T22:12:25.326 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.326 DEBUG:boto:Data: 2023-09-29T22:12:25.326 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.326 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.326 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.327 DEBUG:boto:Params: {} 2023-09-29T22:12:25.327 DEBUG:boto:Token: None 2023-09-29T22:12:25.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005527d873814909d4-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.329 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.329 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.329 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.329 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.329 DEBUG:boto:Method: GET 2023-09-29T22:12:25.329 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.329 DEBUG:boto:Data: 2023-09-29T22:12:25.330 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.330 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.330 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.330 DEBUG:boto:Params: {} 2023-09-29T22:12:25.330 DEBUG:boto:Token: None 2023-09-29T22:12:25.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.330 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e669704a8f04bd0-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.332 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.333 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.333 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.333 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.333 DEBUG:boto:Method: GET 2023-09-29T22:12:25.333 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.333 DEBUG:boto:Data: 2023-09-29T22:12:25.333 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.333 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.333 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.334 DEBUG:boto:Params: {} 2023-09-29T22:12:25.334 DEBUG:boto:Token: None 2023-09-29T22:12:25.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.334 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70ef34170cf8dfb-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.336 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.337 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.337 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.337 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.337 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.337 DEBUG:boto:Method: GET 2023-09-29T22:12:25.337 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.337 DEBUG:boto:Data: 2023-09-29T22:12:25.337 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.338 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.338 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.338 DEBUG:boto:Params: {} 2023-09-29T22:12:25.338 DEBUG:boto:Token: None 2023-09-29T22:12:25.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd61335e1b68ed6-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.341 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.341 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.341 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.341 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.341 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:25.342 DEBUG:boto:Data: 2023-09-29T22:12:25.342 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.342 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.342 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.342 DEBUG:boto:Params: {} 2023-09-29T22:12:25.342 DEBUG:boto:Token: None 2023-09-29T22:12:25.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj 2023-09-29T22:12:25.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1x7qtBsHf5x1fGIAZhG4zkLAnbM= 2023-09-29T22:12:25.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1x7qtBsHf5x1fGIAZhG4zkLAnbM=', 'Content-Length': '0'} 2023-09-29T22:12:25.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d05bd346b7ab8c16-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.344 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.345 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.345 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.345 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:25.345 DEBUG:boto:Data: 2023-09-29T22:12:25.345 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.345 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.345 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.345 DEBUG:boto:Params: {} 2023-09-29T22:12:25.345 DEBUG:boto:Token: None 2023-09-29T22:12:25.346 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj 2023-09-29T22:12:25.346 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1x7qtBsHf5x1fGIAZhG4zkLAnbM= 2023-09-29T22:12:25.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1x7qtBsHf5x1fGIAZhG4zkLAnbM=', 'Content-Length': '0'} 2023-09-29T22:12:25.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed44f468df0426dd-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:25.348 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.348 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.348 DEBUG:boto:Method: GET 2023-09-29T22:12:25.348 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:25.348 DEBUG:boto:Data: 2023-09-29T22:12:25.349 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.349 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.349 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.349 DEBUG:boto:Params: {} 2023-09-29T22:12:25.349 DEBUG:boto:Token: None 2023-09-29T22:12:25.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj 2023-09-29T22:12:25.349 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tzj0Xjuu1B/ygDVt1CdPCcj/PXw= 2023-09-29T22:12:25.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tzj0Xjuu1B/ygDVt1CdPCcj/PXw=', 'Content-Length': '0'} 2023-09-29T22:12:25.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074b671c018f3e175-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.351 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.351 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.352 DEBUG:boto:Method: GET 2023-09-29T22:12:25.352 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:25.352 DEBUG:boto:Data: 2023-09-29T22:12:25.352 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.352 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.352 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.352 DEBUG:boto:Params: {} 2023-09-29T22:12:25.352 DEBUG:boto:Token: None 2023-09-29T22:12:25.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj 2023-09-29T22:12:25.353 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tzj0Xjuu1B/ygDVt1CdPCcj/PXw= 2023-09-29T22:12:25.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tzj0Xjuu1B/ygDVt1CdPCcj/PXw=', 'Content-Length': '0'} 2023-09-29T22:12:25.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c22c575ed1ae1ce-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.355 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.355 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.355 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.355 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.355 DEBUG:boto:Method: GET 2023-09-29T22:12:25.356 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.356 DEBUG:boto:Data: 2023-09-29T22:12:25.356 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.356 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:25.356 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.356 DEBUG:boto:Params: {} 2023-09-29T22:12:25.357 DEBUG:boto:Token: None 2023-09-29T22:12:25.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5630a0c5cc8b87b-0065174bc9-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.359 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.360 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.360 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.360 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.360 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.360 DEBUG:boto:Method: GET 2023-09-29T22:12:25.360 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:25.360 DEBUG:boto:Data: 2023-09-29T22:12:25.360 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.360 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.360 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.361 DEBUG:boto:Params: {} 2023-09-29T22:12:25.361 DEBUG:boto:Token: None 2023-09-29T22:12:25.361 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.361 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005509f18f2b7d5e28-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.406 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-11 zones={test-zone1, test-zone2} 2023-09-29T22:12:25.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:25.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:25.407 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:25.462 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:25.462 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:25.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:25.496+0000 7f607be15880 1 bucket sync caught up with source: 2023-09-29T22:12:25.496 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.41.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.15.6, , ] 2023-09-29T22:12:25.496 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.41.6, , 00000000001.13.6, , , , 00000000001.11.6, 00000000001.15.6, , ] 2023-09-29T22:12:25.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:25.496+0000 7f607be15880 0 bucket checkpoint complete 2023-09-29T22:12:25.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-11 zones={test-zone3, test-zone2} 2023-09-29T22:12:25.507 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.507 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.508 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.508 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:25.508 DEBUG:boto:Data: 2023-09-29T22:12:25.508 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.508 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.508 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.508 DEBUG:boto:Params: {} 2023-09-29T22:12:25.509 DEBUG:boto:Token: None 2023-09-29T22:12:25.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bq59uhE67Epc3n+9nBYVhR35e5Q= 2023-09-29T22:12:25.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bq59uhE67Epc3n+9nBYVhR35e5Q=', 'Content-Length': '0'} 2023-09-29T22:12:25.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066e69a87bfbdb375-0065174bc9-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.511 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.511 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.512 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.512 DEBUG:boto:Path: /dmuhsu-11/ 2023-09-29T22:12:25.512 DEBUG:boto:Data: 2023-09-29T22:12:25.512 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.512 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.512 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.512 DEBUG:boto:Params: {} 2023-09-29T22:12:25.512 DEBUG:boto:Token: None 2023-09-29T22:12:25.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bq59uhE67Epc3n+9nBYVhR35e5Q= 2023-09-29T22:12:25.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bq59uhE67Epc3n+9nBYVhR35e5Q=', 'Content-Length': '0'} 2023-09-29T22:12:25.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f18473211736f468-0065174bc9-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.515 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:25.515 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.515 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.516 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.516 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.516 DEBUG:boto:Method: GET 2023-09-29T22:12:25.516 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.516 DEBUG:boto:Data: 2023-09-29T22:12:25.516 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.516 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.516 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.516 DEBUG:boto:Params: {} 2023-09-29T22:12:25.516 DEBUG:boto:Token: None 2023-09-29T22:12:25.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/?versions 2023-09-29T22:12:25.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao= 2023-09-29T22:12:25.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao=', 'Content-Length': '0'} 2023-09-29T22:12:25.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822ccdec0e559b49-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.519 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:25.520 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:25.520 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.520 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.520 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.520 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.521 DEBUG:boto:Method: GET 2023-09-29T22:12:25.521 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.521 DEBUG:boto:Data: 2023-09-29T22:12:25.521 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.521 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.521 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.521 DEBUG:boto:Params: {} 2023-09-29T22:12:25.521 DEBUG:boto:Token: None 2023-09-29T22:12:25.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/?versions 2023-09-29T22:12:25.521 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao= 2023-09-29T22:12:25.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao=', 'Content-Length': '0'} 2023-09-29T22:12:25.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbc86549d3ecf45-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.524 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.524 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:25.525 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:25.525 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:25.525 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:25.525 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.525 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.525 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.525 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.525 DEBUG:boto:Method: GET 2023-09-29T22:12:25.525 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.526 DEBUG:boto:Data: 2023-09-29T22:12:25.526 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.526 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.526 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.526 DEBUG:boto:Params: {} 2023-09-29T22:12:25.526 DEBUG:boto:Token: None 2023-09-29T22:12:25.526 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/?versions 2023-09-29T22:12:25.526 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao= 2023-09-29T22:12:25.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao=', 'Content-Length': '0'} 2023-09-29T22:12:25.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc92773d7561145-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.529 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.529 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.529 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.530 DEBUG:boto:path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.530 DEBUG:boto:auth_path=/dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.530 DEBUG:boto:Method: GET 2023-09-29T22:12:25.530 DEBUG:boto:Path: /dmuhsu-11/?versions&max-keys=999 2023-09-29T22:12:25.530 DEBUG:boto:Data: 2023-09-29T22:12:25.530 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.530 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.530 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.530 DEBUG:boto:Params: {} 2023-09-29T22:12:25.530 DEBUG:boto:Token: None 2023-09-29T22:12:25.531 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/?versions 2023-09-29T22:12:25.531 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao= 2023-09-29T22:12:25.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xXqp1Is6Cmb4ttIn2N/h2wVlCao=', 'Content-Length': '0'} 2023-09-29T22:12:25.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae733beae4fb530f-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.533 DEBUG:boto:b'dmuhsu-11999false&nulltrue2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:25.534 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.534 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.534 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.534 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.534 DEBUG:boto:Method: GET 2023-09-29T22:12:25.534 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.535 DEBUG:boto:Data: 2023-09-29T22:12:25.535 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.535 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.535 DEBUG:boto:Params: {} 2023-09-29T22:12:25.535 DEBUG:boto:Token: None 2023-09-29T22:12:25.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.535 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc= 2023-09-29T22:12:25.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc=', 'Content-Length': '0'} 2023-09-29T22:12:25.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fa044c1c784e0d2-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.578 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.578 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.579 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.579 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.579 DEBUG:boto:Method: GET 2023-09-29T22:12:25.579 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.579 DEBUG:boto:Data: 2023-09-29T22:12:25.579 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.579 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.579 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.579 DEBUG:boto:Params: {} 2023-09-29T22:12:25.580 DEBUG:boto:Token: None 2023-09-29T22:12:25.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.580 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc= 2023-09-29T22:12:25.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc=', 'Content-Length': '0'} 2023-09-29T22:12:25.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad0ef5bd2e157e55-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.582 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.582 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.582 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.583 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.583 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.583 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.583 DEBUG:boto:Data: 2023-09-29T22:12:25.583 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.583 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.583 DEBUG:boto:Params: {} 2023-09-29T22:12:25.583 DEBUG:boto:Token: None 2023-09-29T22:12:25.583 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UJCyUciFGF0eb1XJdaiJsedM8pY= 2023-09-29T22:12:25.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UJCyUciFGF0eb1XJdaiJsedM8pY=', 'Content-Length': '0'} 2023-09-29T22:12:25.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001193f4b7caff38de-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.586 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.586 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.586 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.586 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.586 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.586 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.586 DEBUG:boto:Data: 2023-09-29T22:12:25.587 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.587 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.587 DEBUG:boto:Params: {} 2023-09-29T22:12:25.587 DEBUG:boto:Token: None 2023-09-29T22:12:25.587 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.587 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UJCyUciFGF0eb1XJdaiJsedM8pY= 2023-09-29T22:12:25.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UJCyUciFGF0eb1XJdaiJsedM8pY=', 'Content-Length': '0'} 2023-09-29T22:12:25.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001597270196a762ba-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:25.589 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.590 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.590 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.590 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.590 DEBUG:boto:Method: GET 2023-09-29T22:12:25.590 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.590 DEBUG:boto:Data: 2023-09-29T22:12:25.590 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.590 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.590 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.591 DEBUG:boto:Params: {} 2023-09-29T22:12:25.591 DEBUG:boto:Token: None 2023-09-29T22:12:25.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc= 2023-09-29T22:12:25.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc=', 'Content-Length': '0'} 2023-09-29T22:12:25.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036e811279fdce788-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.593 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.593 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.593 DEBUG:boto:path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.593 DEBUG:boto:auth_path=/dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.593 DEBUG:boto:Method: GET 2023-09-29T22:12:25.594 DEBUG:boto:Path: /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.594 DEBUG:boto:Data: 2023-09-29T22:12:25.594 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.594 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.594 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.594 DEBUG:boto:Params: {} 2023-09-29T22:12:25.594 DEBUG:boto:Token: None 2023-09-29T22:12:25.594 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26?versionId=null 2023-09-29T22:12:25.594 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc= 2023-09-29T22:12:25.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5nKD2q6iX9xobW3i2yugMaIVpc=', 'Content-Length': '0'} 2023-09-29T22:12:25.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5657bc2ddad569f-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.597 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.597 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.597 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.597 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.597 DEBUG:boto:Method: GET 2023-09-29T22:12:25.597 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.597 DEBUG:boto:Data: 2023-09-29T22:12:25.598 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.598 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.598 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.598 DEBUG:boto:Params: {} 2023-09-29T22:12:25.598 DEBUG:boto:Token: None 2023-09-29T22:12:25.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baafe976fb542e14-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.601 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.601 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.601 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.601 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.601 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.602 DEBUG:boto:Method: GET 2023-09-29T22:12:25.602 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.602 DEBUG:boto:Data: 2023-09-29T22:12:25.602 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.602 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.602 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.602 DEBUG:boto:Params: {} 2023-09-29T22:12:25.602 DEBUG:boto:Token: None 2023-09-29T22:12:25.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.603 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783e2c8a12c54178-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.605 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.606 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.606 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.606 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.606 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:25.606 DEBUG:boto:Data: 2023-09-29T22:12:25.606 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.606 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.606 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.606 DEBUG:boto:Params: {} 2023-09-29T22:12:25.607 DEBUG:boto:Token: None 2023-09-29T22:12:25.607 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26 2023-09-29T22:12:25.607 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hIp/mP/V1/p7/tnPBoqdYUBzyB4= 2023-09-29T22:12:25.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hIp/mP/V1/p7/tnPBoqdYUBzyB4=', 'Content-Length': '0'} 2023-09-29T22:12:25.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000304276f876184221-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.609 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.609 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.609 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.609 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:25.609 DEBUG:boto:Data: 2023-09-29T22:12:25.609 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.609 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.609 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.610 DEBUG:boto:Params: {} 2023-09-29T22:12:25.610 DEBUG:boto:Token: None 2023-09-29T22:12:25.610 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26 2023-09-29T22:12:25.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hIp/mP/V1/p7/tnPBoqdYUBzyB4= 2023-09-29T22:12:25.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hIp/mP/V1/p7/tnPBoqdYUBzyB4=', 'Content-Length': '0'} 2023-09-29T22:12:25.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e9abc37a70b81a4-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:25.612 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.612 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.612 DEBUG:boto:Method: GET 2023-09-29T22:12:25.613 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:25.613 DEBUG:boto:Data: 2023-09-29T22:12:25.613 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.613 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.613 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.613 DEBUG:boto:Params: {} 2023-09-29T22:12:25.613 DEBUG:boto:Token: None 2023-09-29T22:12:25.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26 2023-09-29T22:12:25.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zYJaWVj8TiZNfpNSGKLnViYQtZI= 2023-09-29T22:12:25.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zYJaWVj8TiZNfpNSGKLnViYQtZI=', 'Content-Length': '0'} 2023-09-29T22:12:25.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a68eaab9269cbc82-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.615 DEBUG:boto:path=/dmuhsu-11/%26 2023-09-29T22:12:25.615 DEBUG:boto:auth_path=/dmuhsu-11/%26 2023-09-29T22:12:25.616 DEBUG:boto:Method: GET 2023-09-29T22:12:25.616 DEBUG:boto:Path: /dmuhsu-11/%26 2023-09-29T22:12:25.616 DEBUG:boto:Data: 2023-09-29T22:12:25.616 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.616 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.616 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.616 DEBUG:boto:Params: {} 2023-09-29T22:12:25.616 DEBUG:boto:Token: None 2023-09-29T22:12:25.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%26 2023-09-29T22:12:25.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zYJaWVj8TiZNfpNSGKLnViYQtZI= 2023-09-29T22:12:25.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zYJaWVj8TiZNfpNSGKLnViYQtZI=', 'Content-Length': '0'} 2023-09-29T22:12:25.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000778720483296314e-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.619 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.619 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.619 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.619 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.619 DEBUG:boto:Method: GET 2023-09-29T22:12:25.619 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.619 DEBUG:boto:Data: 2023-09-29T22:12:25.619 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.619 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.620 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.620 DEBUG:boto:Params: {} 2023-09-29T22:12:25.620 DEBUG:boto:Token: None 2023-09-29T22:12:25.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3b3e5966082eed-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.622 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.623 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.623 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.623 DEBUG:boto:path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.623 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.623 DEBUG:boto:Method: GET 2023-09-29T22:12:25.623 DEBUG:boto:Path: /dmuhsu-11/?prefix=%26 2023-09-29T22:12:25.623 DEBUG:boto:Data: 2023-09-29T22:12:25.623 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.623 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.624 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.624 DEBUG:boto:Params: {} 2023-09-29T22:12:25.624 DEBUG:boto:Token: None 2023-09-29T22:12:25.624 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe923115e59899e-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.670 DEBUG:boto:b'dmuhsu-11&1000false&2023-09-29T22:12:07.858Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:25.671 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.671 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.671 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.671 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.671 DEBUG:boto:Method: GET 2023-09-29T22:12:25.672 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.672 DEBUG:boto:Data: 2023-09-29T22:12:25.672 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.672 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.672 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.672 DEBUG:boto:Params: {} 2023-09-29T22:12:25.672 DEBUG:boto:Token: None 2023-09-29T22:12:25.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fb96dfbeee31737-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.675 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.675 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.675 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.675 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.675 DEBUG:boto:Method: GET 2023-09-29T22:12:25.675 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.675 DEBUG:boto:Data: 2023-09-29T22:12:25.675 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.676 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.676 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.676 DEBUG:boto:Params: {} 2023-09-29T22:12:25.676 DEBUG:boto:Token: None 2023-09-29T22:12:25.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f2af01e7b4a6c87-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.678 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.678 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.679 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.679 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.679 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.679 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.679 DEBUG:boto:Data: 2023-09-29T22:12:25.679 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.679 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.679 DEBUG:boto:Params: {} 2023-09-29T22:12:25.680 DEBUG:boto:Token: None 2023-09-29T22:12:25.680 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8= 2023-09-29T22:12:25.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8=', 'Content-Length': '0'} 2023-09-29T22:12:25.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b247846021900e9-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.681 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.682 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.682 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.682 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.682 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.682 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.682 DEBUG:boto:Data: 2023-09-29T22:12:25.682 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.682 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.683 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.683 DEBUG:boto:Params: {} 2023-09-29T22:12:25.683 DEBUG:boto:Token: None 2023-09-29T22:12:25.683 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.683 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8= 2023-09-29T22:12:25.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HovCk6CU0YmCJkaa1BdubGp98G8=', 'Content-Length': '0'} 2023-09-29T22:12:25.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f85ed1e050e3cd6-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:25.685 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.685 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.685 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.685 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.686 DEBUG:boto:Method: GET 2023-09-29T22:12:25.686 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.686 DEBUG:boto:Data: 2023-09-29T22:12:25.686 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.686 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.686 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.686 DEBUG:boto:Params: {} 2023-09-29T22:12:25.686 DEBUG:boto:Token: None 2023-09-29T22:12:25.687 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8c08dead04bc629-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.730 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.731 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.731 DEBUG:boto:path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.731 DEBUG:boto:auth_path=/dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.731 DEBUG:boto:Method: GET 2023-09-29T22:12:25.731 DEBUG:boto:Path: /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.731 DEBUG:boto:Data: 2023-09-29T22:12:25.731 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.731 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.732 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.732 DEBUG:boto:Params: {} 2023-09-29T22:12:25.732 DEBUG:boto:Token: None 2023-09-29T22:12:25.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A?versionId=null 2023-09-29T22:12:25.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4= 2023-09-29T22:12:25.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y2c3DBCoTzXXm7BD5C9XYh5Qtn4=', 'Content-Length': '0'} 2023-09-29T22:12:25.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000837743b38820bec7-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.735 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.735 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.735 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.735 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.735 DEBUG:boto:Method: GET 2023-09-29T22:12:25.735 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.735 DEBUG:boto:Data: 2023-09-29T22:12:25.735 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.736 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.736 DEBUG:boto:Params: {} 2023-09-29T22:12:25.736 DEBUG:boto:Token: None 2023-09-29T22:12:25.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009829440119bc92e8-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.738 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.739 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.739 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.739 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.739 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.739 DEBUG:boto:Method: GET 2023-09-29T22:12:25.739 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.740 DEBUG:boto:Data: 2023-09-29T22:12:25.740 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.740 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.740 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.740 DEBUG:boto:Params: {} 2023-09-29T22:12:25.740 DEBUG:boto:Token: None 2023-09-29T22:12:25.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014097945a3cbf4ca-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.743 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.743 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.744 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.744 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.744 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.744 DEBUG:boto:Data: 2023-09-29T22:12:25.744 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.744 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.744 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.744 DEBUG:boto:Params: {} 2023-09-29T22:12:25.744 DEBUG:boto:Token: None 2023-09-29T22:12:25.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A= 2023-09-29T22:12:25.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A=', 'Content-Length': '0'} 2023-09-29T22:12:25.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d219831bccd5b4-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.746 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.747 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.747 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.747 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.747 DEBUG:boto:Data: 2023-09-29T22:12:25.747 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.747 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.747 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.747 DEBUG:boto:Params: {} 2023-09-29T22:12:25.748 DEBUG:boto:Token: None 2023-09-29T22:12:25.748 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A= 2023-09-29T22:12:25.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dRcfxs0iDSzo1jAoRf6PxdsFB/A=', 'Content-Length': '0'} 2023-09-29T22:12:25.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0c7d00f793d2790-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:25.750 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.750 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.750 DEBUG:boto:Method: GET 2023-09-29T22:12:25.753 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.753 DEBUG:boto:Data: 2023-09-29T22:12:25.753 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.753 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.753 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.754 DEBUG:boto:Params: {} 2023-09-29T22:12:25.754 DEBUG:boto:Token: None 2023-09-29T22:12:25.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.754 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E= 2023-09-29T22:12:25.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E=', 'Content-Length': '0'} 2023-09-29T22:12:25.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c0dcadf0acf1ca9-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.798 DEBUG:boto:path=/dmuhsu-11/%3A 2023-09-29T22:12:25.798 DEBUG:boto:auth_path=/dmuhsu-11/%3A 2023-09-29T22:12:25.799 DEBUG:boto:Method: GET 2023-09-29T22:12:25.799 DEBUG:boto:Path: /dmuhsu-11/%3A 2023-09-29T22:12:25.799 DEBUG:boto:Data: 2023-09-29T22:12:25.799 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.799 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.799 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.799 DEBUG:boto:Params: {} 2023-09-29T22:12:25.799 DEBUG:boto:Token: None 2023-09-29T22:12:25.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/%3A 2023-09-29T22:12:25.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E= 2023-09-29T22:12:25.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ciKuBlAawG08wAdJOvImUJ5gR6E=', 'Content-Length': '0'} 2023-09-29T22:12:25.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e1f3da231935f9d-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.802 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.802 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.802 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.802 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.802 DEBUG:boto:Method: GET 2023-09-29T22:12:25.802 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.803 DEBUG:boto:Data: 2023-09-29T22:12:25.803 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.803 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.803 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.803 DEBUG:boto:Params: {} 2023-09-29T22:12:25.803 DEBUG:boto:Token: None 2023-09-29T22:12:25.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.803 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002be2014155fc63e-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.806 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.806 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.806 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.806 DEBUG:boto:path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.806 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.806 DEBUG:boto:Method: GET 2023-09-29T22:12:25.807 DEBUG:boto:Path: /dmuhsu-11/?prefix=%3A 2023-09-29T22:12:25.807 DEBUG:boto:Data: 2023-09-29T22:12:25.807 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.807 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.807 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.807 DEBUG:boto:Params: {} 2023-09-29T22:12:25.807 DEBUG:boto:Token: None 2023-09-29T22:12:25.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.807 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e5b5cecc6fd0e5-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.810 DEBUG:boto:b'dmuhsu-11:1000false:2023-09-29T22:12:07.808Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.810 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.811 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.811 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.811 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.811 DEBUG:boto:Method: GET 2023-09-29T22:12:25.811 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.811 DEBUG:boto:Data: 2023-09-29T22:12:25.811 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.811 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.811 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.811 DEBUG:boto:Params: {} 2023-09-29T22:12:25.812 DEBUG:boto:Token: None 2023-09-29T22:12:25.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015b4ae013b4568d9-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.814 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.814 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.814 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.814 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.814 DEBUG:boto:Method: GET 2023-09-29T22:12:25.814 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.814 DEBUG:boto:Data: 2023-09-29T22:12:25.815 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.815 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.815 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.815 DEBUG:boto:Params: {} 2023-09-29T22:12:25.815 DEBUG:boto:Token: None 2023-09-29T22:12:25.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.815 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bb18fdc3df27dd4-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.817 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.817 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.818 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.818 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.818 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.818 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.818 DEBUG:boto:Data: 2023-09-29T22:12:25.818 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.818 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.818 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.818 DEBUG:boto:Params: {} 2023-09-29T22:12:25.819 DEBUG:boto:Token: None 2023-09-29T22:12:25.819 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.819 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA= 2023-09-29T22:12:25.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA=', 'Content-Length': '0'} 2023-09-29T22:12:25.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d08913046ebdb4-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.821 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.821 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.821 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.821 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.821 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.821 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.821 DEBUG:boto:Data: 2023-09-29T22:12:25.821 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.821 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.821 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.822 DEBUG:boto:Params: {} 2023-09-29T22:12:25.822 DEBUG:boto:Token: None 2023-09-29T22:12:25.822 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.822 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA= 2023-09-29T22:12:25.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8WpZ5MwgtdSU0aPh1RMwwluHBgA=', 'Content-Length': '0'} 2023-09-29T22:12:25.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004972e61686f557f0-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.824 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.824 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.824 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.824 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.825 DEBUG:boto:Method: GET 2023-09-29T22:12:25.825 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.825 DEBUG:boto:Data: 2023-09-29T22:12:25.825 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.825 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.825 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.825 DEBUG:boto:Params: {} 2023-09-29T22:12:25.825 DEBUG:boto:Token: None 2023-09-29T22:12:25.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.825 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000778581a9ddf33bfb-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.827 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.828 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.828 DEBUG:boto:path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.828 DEBUG:boto:auth_path=/dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.828 DEBUG:boto:Method: GET 2023-09-29T22:12:25.828 DEBUG:boto:Path: /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.828 DEBUG:boto:Data: 2023-09-29T22:12:25.828 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.828 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.828 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.829 DEBUG:boto:Params: {} 2023-09-29T22:12:25.829 DEBUG:boto:Token: None 2023-09-29T22:12:25.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj?versionId=null 2023-09-29T22:12:25.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA= 2023-09-29T22:12:25.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:axr7GpAgCno2slRoTkjHpT8/gWA=', 'Content-Length': '0'} 2023-09-29T22:12:25.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad047f7650893a6a-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.831 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.831 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.831 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.831 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.831 DEBUG:boto:Method: GET 2023-09-29T22:12:25.832 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.832 DEBUG:boto:Data: 2023-09-29T22:12:25.832 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.832 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.832 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.832 DEBUG:boto:Params: {} 2023-09-29T22:12:25.832 DEBUG:boto:Token: None 2023-09-29T22:12:25.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059a1f3a64880042a-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.835 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.835 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.835 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.835 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.835 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.835 DEBUG:boto:Method: GET 2023-09-29T22:12:25.835 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.836 DEBUG:boto:Data: 2023-09-29T22:12:25.836 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.836 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.836 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.836 DEBUG:boto:Params: {} 2023-09-29T22:12:25.836 DEBUG:boto:Token: None 2023-09-29T22:12:25.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003796a005a56e100d-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.839 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.839 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.839 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.839 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.840 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.840 DEBUG:boto:Data: 2023-09-29T22:12:25.840 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.840 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.840 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.840 DEBUG:boto:Params: {} 2023-09-29T22:12:25.840 DEBUG:boto:Token: None 2023-09-29T22:12:25.840 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU= 2023-09-29T22:12:25.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU=', 'Content-Length': '0'} 2023-09-29T22:12:25.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009739eca95adb8c28-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.842 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.842 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.842 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.843 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.843 DEBUG:boto:Data: 2023-09-29T22:12:25.843 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.843 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.843 DEBUG:boto:Params: {} 2023-09-29T22:12:25.843 DEBUG:boto:Token: None 2023-09-29T22:12:25.843 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.843 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU= 2023-09-29T22:12:25.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIIlqKDYo3qc2UDh0yvrYPEhDQU=', 'Content-Length': '0'} 2023-09-29T22:12:25.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb6cd6d43f6858e6-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:25.845 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.846 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.846 DEBUG:boto:Method: GET 2023-09-29T22:12:25.846 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.846 DEBUG:boto:Data: 2023-09-29T22:12:25.846 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.846 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.846 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.846 DEBUG:boto:Params: {} 2023-09-29T22:12:25.846 DEBUG:boto:Token: None 2023-09-29T22:12:25.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI= 2023-09-29T22:12:25.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI=', 'Content-Length': '0'} 2023-09-29T22:12:25.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d3bd0538ee7a11b-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.890 DEBUG:boto:path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.890 DEBUG:boto:auth_path=/dmuhsu-11/_myobj 2023-09-29T22:12:25.891 DEBUG:boto:Method: GET 2023-09-29T22:12:25.891 DEBUG:boto:Path: /dmuhsu-11/_myobj 2023-09-29T22:12:25.891 DEBUG:boto:Data: 2023-09-29T22:12:25.891 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.891 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.891 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.891 DEBUG:boto:Params: {} 2023-09-29T22:12:25.892 DEBUG:boto:Token: None 2023-09-29T22:12:25.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/_myobj 2023-09-29T22:12:25.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI= 2023-09-29T22:12:25.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipY5aCod7wxhXlCV+JzdFV/5XrI=', 'Content-Length': '0'} 2023-09-29T22:12:25.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfeb532a65979bd5-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.894 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.894 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.894 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.894 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.895 DEBUG:boto:Method: GET 2023-09-29T22:12:25.895 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.895 DEBUG:boto:Data: 2023-09-29T22:12:25.895 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.895 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.895 DEBUG:boto:Params: {} 2023-09-29T22:12:25.895 DEBUG:boto:Token: None 2023-09-29T22:12:25.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd8dfee47488720-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.898 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.899 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:25.899 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:25.899 DEBUG:boto:path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.899 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.899 DEBUG:boto:Method: GET 2023-09-29T22:12:25.900 DEBUG:boto:Path: /dmuhsu-11/?prefix=_myobj 2023-09-29T22:12:25.900 DEBUG:boto:Data: 2023-09-29T22:12:25.900 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.900 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.900 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.900 DEBUG:boto:Params: {} 2023-09-29T22:12:25.900 DEBUG:boto:Token: None 2023-09-29T22:12:25.901 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/ 2023-09-29T22:12:25.901 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA= 2023-09-29T22:12:25.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rV/lII86wwAplpkPiAJ075XjOjA=', 'Content-Length': '0'} 2023-09-29T22:12:25.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d508500ed57be12-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.903 DEBUG:boto:b'dmuhsu-11_myobj1000false_myobj2023-09-29T22:12:07.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:25.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:25.904 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.904 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.904 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.904 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.904 DEBUG:boto:Method: GET 2023-09-29T22:12:25.905 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.905 DEBUG:boto:Data: 2023-09-29T22:12:25.905 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.905 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.905 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.905 DEBUG:boto:Params: {} 2023-09-29T22:12:25.905 DEBUG:boto:Token: None 2023-09-29T22:12:25.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.906 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009205198b518e2625-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.908 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.908 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.908 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.908 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.908 DEBUG:boto:Method: GET 2023-09-29T22:12:25.909 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.909 DEBUG:boto:Data: 2023-09-29T22:12:25.909 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.909 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.909 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.909 DEBUG:boto:Params: {} 2023-09-29T22:12:25.909 DEBUG:boto:Token: None 2023-09-29T22:12:25.909 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.910 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007583b1325bbfd0e1-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.954 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.955 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.955 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.955 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.955 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.955 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.955 DEBUG:boto:Data: 2023-09-29T22:12:25.955 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.956 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.956 DEBUG:boto:Params: {} 2023-09-29T22:12:25.956 DEBUG:boto:Token: None 2023-09-29T22:12:25.956 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo= 2023-09-29T22:12:25.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo=', 'Content-Length': '0'} 2023-09-29T22:12:25.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017a3a92b94e082f8-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.959 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.961 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.961 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.961 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.961 DEBUG:boto:Method: HEAD 2023-09-29T22:12:25.962 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.962 DEBUG:boto:Data: 2023-09-29T22:12:25.962 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.962 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:25.962 DEBUG:boto:Port: 8000 2023-09-29T22:12:25.962 DEBUG:boto:Params: {} 2023-09-29T22:12:25.962 DEBUG:boto:Token: None 2023-09-29T22:12:25.963 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.963 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo= 2023-09-29T22:12:25.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BEWkH7+NalQnozIpRArHutSmojo=', 'Content-Length': '0'} 2023-09-29T22:12:25.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007c049e32364ce37-0065174bc9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:25.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:25.965 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:25.965 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:25.965 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.965 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.966 DEBUG:boto:Method: GET 2023-09-29T22:12:25.966 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.966 DEBUG:boto:Data: 2023-09-29T22:12:25.966 DEBUG:boto:Headers: {} 2023-09-29T22:12:25.966 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:25.966 DEBUG:boto:Port: 8001 2023-09-29T22:12:25.966 DEBUG:boto:Params: {} 2023-09-29T22:12:25.966 DEBUG:boto:Token: None 2023-09-29T22:12:25.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:25 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:25.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA= 2023-09-29T22:12:25.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p4AXt2xepc83Rcc1o4i08pcuBvA=', 'Content-Length': '0'} 2023-09-29T22:12:25.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe44737e86e6f503-0065174bc9-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.010 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:26.010 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:26.011 DEBUG:boto:path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:26.011 DEBUG:boto:auth_path=/dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:26.011 DEBUG:boto:Method: GET 2023-09-29T22:12:26.011 DEBUG:boto:Path: /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:26.011 DEBUG:boto:Data: 2023-09-29T22:12:26.011 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.011 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:26.011 DEBUG:boto:Port: 8000 2023-09-29T22:12:26.011 DEBUG:boto:Params: {} 2023-09-29T22:12:26.012 DEBUG:boto:Token: None 2023-09-29T22:12:26.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/myobj?versionId=null 2023-09-29T22:12:26.012 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Be875Qx8k62pyYsrCfAM2E0RCLQ= 2023-09-29T22:12:26.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Be875Qx8k62pyYsrCfAM2E0RCLQ=', 'Content-Length': '0'} 2023-09-29T22:12:26.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ae3a732445e8fec-0065174bca-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.014 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:26.014 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:26.014 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.015 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.015 DEBUG:boto:Method: GET 2023-09-29T22:12:26.015 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.015 DEBUG:boto:Data: 2023-09-29T22:12:26.015 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:26.015 DEBUG:boto:Port: 8001 2023-09-29T22:12:26.015 DEBUG:boto:Params: {} 2023-09-29T22:12:26.015 DEBUG:boto:Token: None 2023-09-29T22:12:26.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/ 2023-09-29T22:12:26.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E= 2023-09-29T22:12:26.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E=', 'Content-Length': '0'} 2023-09-29T22:12:26.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63833e82ed266ee-0065174bca-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.018 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:26.018 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:26.019 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:26.019 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.019 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.019 DEBUG:boto:Method: GET 2023-09-29T22:12:26.019 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.019 DEBUG:boto:Data: 2023-09-29T22:12:26.019 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:26.019 DEBUG:boto:Port: 8000 2023-09-29T22:12:26.019 DEBUG:boto:Params: {} 2023-09-29T22:12:26.020 DEBUG:boto:Token: None 2023-09-29T22:12:26.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/ 2023-09-29T22:12:26.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E= 2023-09-29T22:12:26.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E=', 'Content-Length': '0'} 2023-09-29T22:12:26.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fa49af24103df0-0065174bca-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.023 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:26.023 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:26.023 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:26.023 DEBUG:boto:Method: HEAD 2023-09-29T22:12:26.023 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:26.023 DEBUG:boto:Data: 2023-09-29T22:12:26.023 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.024 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:26.024 DEBUG:boto:Port: 8001 2023-09-29T22:12:26.024 DEBUG:boto:Params: {} 2023-09-29T22:12:26.024 DEBUG:boto:Token: None 2023-09-29T22:12:26.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/myobj 2023-09-29T22:12:26.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ywub/y81qyIRvGWaUrXRzw6Eh9Q= 2023-09-29T22:12:26.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ywub/y81qyIRvGWaUrXRzw6Eh9Q=', 'Content-Length': '0'} 2023-09-29T22:12:26.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039f224a2a8bebeda-0065174bca-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.026 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:26.026 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:26.026 DEBUG:boto:Method: HEAD 2023-09-29T22:12:26.027 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:26.027 DEBUG:boto:Data: 2023-09-29T22:12:26.027 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.027 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:26.027 DEBUG:boto:Port: 8000 2023-09-29T22:12:26.027 DEBUG:boto:Params: {} 2023-09-29T22:12:26.027 DEBUG:boto:Token: None 2023-09-29T22:12:26.027 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/myobj 2023-09-29T22:12:26.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ywub/y81qyIRvGWaUrXRzw6Eh9Q= 2023-09-29T22:12:26.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ywub/y81qyIRvGWaUrXRzw6Eh9Q=', 'Content-Length': '0'} 2023-09-29T22:12:26.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002bc4443a15a8a77-0065174bca-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:26.030 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:26.030 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:26.030 DEBUG:boto:Method: GET 2023-09-29T22:12:26.030 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:26.030 DEBUG:boto:Data: 2023-09-29T22:12:26.030 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:26.030 DEBUG:boto:Port: 8001 2023-09-29T22:12:26.031 DEBUG:boto:Params: {} 2023-09-29T22:12:26.031 DEBUG:boto:Token: None 2023-09-29T22:12:26.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/myobj 2023-09-29T22:12:26.031 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4p2juZ2RrhZkZcI5yrtfwbXi2Ew= 2023-09-29T22:12:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4p2juZ2RrhZkZcI5yrtfwbXi2Ew=', 'Content-Length': '0'} 2023-09-29T22:12:26.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdd287d59a6095c2-0065174bca-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.033 DEBUG:boto:path=/dmuhsu-11/myobj 2023-09-29T22:12:26.033 DEBUG:boto:auth_path=/dmuhsu-11/myobj 2023-09-29T22:12:26.033 DEBUG:boto:Method: GET 2023-09-29T22:12:26.033 DEBUG:boto:Path: /dmuhsu-11/myobj 2023-09-29T22:12:26.033 DEBUG:boto:Data: 2023-09-29T22:12:26.033 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.033 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:26.034 DEBUG:boto:Port: 8000 2023-09-29T22:12:26.034 DEBUG:boto:Params: {} 2023-09-29T22:12:26.034 DEBUG:boto:Token: None 2023-09-29T22:12:26.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/myobj 2023-09-29T22:12:26.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4p2juZ2RrhZkZcI5yrtfwbXi2Ew= 2023-09-29T22:12:26.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4p2juZ2RrhZkZcI5yrtfwbXi2Ew=', 'Content-Length': '0'} 2023-09-29T22:12:26.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dccdf65538be0d5-0065174bca-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.036 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:26.036 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:26.036 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.037 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.037 DEBUG:boto:Method: GET 2023-09-29T22:12:26.037 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.037 DEBUG:boto:Data: 2023-09-29T22:12:26.037 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:26.037 DEBUG:boto:Port: 8001 2023-09-29T22:12:26.037 DEBUG:boto:Params: {} 2023-09-29T22:12:26.037 DEBUG:boto:Token: None 2023-09-29T22:12:26.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/ 2023-09-29T22:12:26.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E= 2023-09-29T22:12:26.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E=', 'Content-Length': '0'} 2023-09-29T22:12:26.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf03c6e425fcad0-0065174bca-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.040 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:26.040 DEBUG:boto:path=/dmuhsu-11/ 2023-09-29T22:12:26.040 DEBUG:boto:auth_path=/dmuhsu-11/ 2023-09-29T22:12:26.041 DEBUG:boto:path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.041 DEBUG:boto:auth_path=/dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.041 DEBUG:boto:Method: GET 2023-09-29T22:12:26.041 DEBUG:boto:Path: /dmuhsu-11/?prefix=myobj 2023-09-29T22:12:26.041 DEBUG:boto:Data: 2023-09-29T22:12:26.041 DEBUG:boto:Headers: {} 2023-09-29T22:12:26.041 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:26.041 DEBUG:boto:Port: 8000 2023-09-29T22:12:26.041 DEBUG:boto:Params: {} 2023-09-29T22:12:26.042 DEBUG:boto:Token: None 2023-09-29T22:12:26.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:26 GMT /dmuhsu-11/ 2023-09-29T22:12:26.042 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E= 2023-09-29T22:12:26.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EOzK6OXT3XBvXKDdDCrGtQkiB0E=', 'Content-Length': '0'} 2023-09-29T22:12:26.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ddd88f2bf1fb897-0065174bca-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:26.044 DEBUG:boto:b'dmuhsu-11myobj1000falsemyobj2023-09-29T22:12:07.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:26.045 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-11 zones={test-zone3, test-zone2} 2023-09-29T22:12:26.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:26.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:26.045 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:26.098 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.098 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:26.131+0000 7f3b3069d880 1 waiting to reach incremental sync.. 2023-09-29T22:12:31.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:31.132+0000 7f3b3069d880 1 waiting to reach incremental sync.. 2023-09-29T22:12:36.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:36.133+0000 7f3b3069d880 1 bucket sync caught up with source: 2023-09-29T22:12:36.133 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.10.6, , 00000000001.18.6, , , , 00000000001.27.6, 00000000001.11.6, , ] 2023-09-29T22:12:36.133 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.10.6, , 00000000001.18.6, , , , 00000000001.27.6, 00000000001.11.6, , ] 2023-09-29T22:12:36.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:36.133+0000 7f3b3069d880 0 bucket checkpoint complete 2023-09-29T22:12:36.148 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-12 zones={test-zone1, test-zone3} 2023-09-29T22:12:36.148 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.148 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.148 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.149 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:36.149 DEBUG:boto:Data: 2023-09-29T22:12:36.149 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.149 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.149 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.149 DEBUG:boto:Params: {} 2023-09-29T22:12:36.149 DEBUG:boto:Token: None 2023-09-29T22:12:36.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.150 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78= 2023-09-29T22:12:36.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78=', 'Content-Length': '0'} 2023-09-29T22:12:36.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d651b7c7c4c705a-0065174bd4-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.152 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.152 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.152 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.152 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:36.153 DEBUG:boto:Data: 2023-09-29T22:12:36.153 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.153 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.153 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.153 DEBUG:boto:Params: {} 2023-09-29T22:12:36.153 DEBUG:boto:Token: None 2023-09-29T22:12:36.153 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.153 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78= 2023-09-29T22:12:36.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78=', 'Content-Length': '0'} 2023-09-29T22:12:36.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ae3007b0edeca6f-0065174bd4-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:36.156 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.156 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.156 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.156 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.156 DEBUG:boto:Method: GET 2023-09-29T22:12:36.156 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.156 DEBUG:boto:Data: 2023-09-29T22:12:36.156 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.157 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.157 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.157 DEBUG:boto:Params: {} 2023-09-29T22:12:36.157 DEBUG:boto:Token: None 2023-09-29T22:12:36.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.157 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1fe8c3d355b8afd-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.160 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.160 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:36.161 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:36.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:36.161 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.161 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.161 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.161 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.161 DEBUG:boto:Method: GET 2023-09-29T22:12:36.162 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.162 DEBUG:boto:Data: 2023-09-29T22:12:36.162 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.162 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.162 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.162 DEBUG:boto:Params: {} 2023-09-29T22:12:36.162 DEBUG:boto:Token: None 2023-09-29T22:12:36.162 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.162 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d7dbad85d316f5-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.165 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.165 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:36.165 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:36.166 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:36.166 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:36.166 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.166 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.166 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.166 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.166 DEBUG:boto:Method: GET 2023-09-29T22:12:36.166 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.166 DEBUG:boto:Data: 2023-09-29T22:12:36.167 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.167 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.167 DEBUG:boto:Params: {} 2023-09-29T22:12:36.167 DEBUG:boto:Token: None 2023-09-29T22:12:36.167 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.167 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a729f454cc5544-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.169 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.170 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.170 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.170 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.170 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.170 DEBUG:boto:Method: GET 2023-09-29T22:12:36.170 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.171 DEBUG:boto:Data: 2023-09-29T22:12:36.171 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.171 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.171 DEBUG:boto:Params: {} 2023-09-29T22:12:36.171 DEBUG:boto:Token: None 2023-09-29T22:12:36.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.171 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009009c2162125bba5-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.174 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.174 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.175 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.175 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.175 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.175 DEBUG:boto:Method: GET 2023-09-29T22:12:36.175 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.175 DEBUG:boto:Data: 2023-09-29T22:12:36.175 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.175 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.175 DEBUG:boto:Params: {} 2023-09-29T22:12:36.176 DEBUG:boto:Token: None 2023-09-29T22:12:36.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bdbe12cd1c9deb4-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.219 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.219 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.219 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.219 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.219 DEBUG:boto:Method: GET 2023-09-29T22:12:36.219 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.219 DEBUG:boto:Data: 2023-09-29T22:12:36.219 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.220 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.220 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.220 DEBUG:boto:Params: {} 2023-09-29T22:12:36.220 DEBUG:boto:Token: None 2023-09-29T22:12:36.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000357b641299e11f00-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.222 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.223 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.223 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.223 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.223 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.223 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.223 DEBUG:boto:Data: 2023-09-29T22:12:36.223 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.223 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.223 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.223 DEBUG:boto:Params: {} 2023-09-29T22:12:36.224 DEBUG:boto:Token: None 2023-09-29T22:12:36.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM= 2023-09-29T22:12:36.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM=', 'Content-Length': '0'} 2023-09-29T22:12:36.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2768f9841c4566c-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.226 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.226 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.226 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.226 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.226 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.226 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.226 DEBUG:boto:Data: 2023-09-29T22:12:36.226 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.227 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.227 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.227 DEBUG:boto:Params: {} 2023-09-29T22:12:36.227 DEBUG:boto:Token: None 2023-09-29T22:12:36.227 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.227 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM= 2023-09-29T22:12:36.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM=', 'Content-Length': '0'} 2023-09-29T22:12:36.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd0406a3729308f4-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.229 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.229 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.229 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.229 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.230 DEBUG:boto:Method: GET 2023-09-29T22:12:36.230 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.230 DEBUG:boto:Data: 2023-09-29T22:12:36.230 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.230 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.230 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.230 DEBUG:boto:Params: {} 2023-09-29T22:12:36.230 DEBUG:boto:Token: None 2023-09-29T22:12:36.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.231 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000662b076f8d3fe5c5-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.233 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.233 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.233 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.233 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.233 DEBUG:boto:Method: GET 2023-09-29T22:12:36.233 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.233 DEBUG:boto:Data: 2023-09-29T22:12:36.233 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.233 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.234 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.234 DEBUG:boto:Params: {} 2023-09-29T22:12:36.234 DEBUG:boto:Token: None 2023-09-29T22:12:36.234 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.234 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c768eb5c54c7799f-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.236 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.236 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.236 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.236 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.236 DEBUG:boto:Method: GET 2023-09-29T22:12:36.237 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.237 DEBUG:boto:Data: 2023-09-29T22:12:36.237 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.237 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.237 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.237 DEBUG:boto:Params: {} 2023-09-29T22:12:36.237 DEBUG:boto:Token: None 2023-09-29T22:12:36.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e866ac5eb7e5ef55-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.240 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.240 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.241 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.241 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.241 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.241 DEBUG:boto:Method: GET 2023-09-29T22:12:36.241 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.241 DEBUG:boto:Data: 2023-09-29T22:12:36.241 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.241 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.241 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.241 DEBUG:boto:Params: {} 2023-09-29T22:12:36.242 DEBUG:boto:Token: None 2023-09-29T22:12:36.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.242 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f0f82886a7d0bb-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.244 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.245 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.245 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.245 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.245 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.245 DEBUG:boto:Data: 2023-09-29T22:12:36.245 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.245 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.246 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.246 DEBUG:boto:Params: {} 2023-09-29T22:12:36.246 DEBUG:boto:Token: None 2023-09-29T22:12:36.246 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.246 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY= 2023-09-29T22:12:36.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY=', 'Content-Length': '0'} 2023-09-29T22:12:36.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094a4ff6ce6690085-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.248 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.248 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.248 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.248 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.248 DEBUG:boto:Data: 2023-09-29T22:12:36.248 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.248 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.249 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.249 DEBUG:boto:Params: {} 2023-09-29T22:12:36.249 DEBUG:boto:Token: None 2023-09-29T22:12:36.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.249 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY= 2023-09-29T22:12:36.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY=', 'Content-Length': '0'} 2023-09-29T22:12:36.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006169c21546e4c6e-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.251 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.251 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.251 DEBUG:boto:Method: GET 2023-09-29T22:12:36.251 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.251 DEBUG:boto:Data: 2023-09-29T22:12:36.251 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.252 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.252 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.252 DEBUG:boto:Params: {} 2023-09-29T22:12:36.252 DEBUG:boto:Token: None 2023-09-29T22:12:36.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE= 2023-09-29T22:12:36.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE=', 'Content-Length': '0'} 2023-09-29T22:12:36.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a86b5bb79f0cdbd-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.254 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.254 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.254 DEBUG:boto:Method: GET 2023-09-29T22:12:36.254 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.255 DEBUG:boto:Data: 2023-09-29T22:12:36.255 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.255 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.255 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.255 DEBUG:boto:Params: {} 2023-09-29T22:12:36.255 DEBUG:boto:Token: None 2023-09-29T22:12:36.255 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.255 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE= 2023-09-29T22:12:36.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE=', 'Content-Length': '0'} 2023-09-29T22:12:36.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007557f9967027e99c-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.298 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.299 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.299 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.299 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.299 DEBUG:boto:Method: GET 2023-09-29T22:12:36.299 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.299 DEBUG:boto:Data: 2023-09-29T22:12:36.299 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.299 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.299 DEBUG:boto:Params: {} 2023-09-29T22:12:36.300 DEBUG:boto:Token: None 2023-09-29T22:12:36.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.300 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000182f0ede92368245-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.302 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.303 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.303 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.303 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.303 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.303 DEBUG:boto:Method: GET 2023-09-29T22:12:36.303 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.303 DEBUG:boto:Data: 2023-09-29T22:12:36.303 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.303 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.304 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.304 DEBUG:boto:Params: {} 2023-09-29T22:12:36.304 DEBUG:boto:Token: None 2023-09-29T22:12:36.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6eabf265fc8103-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.306 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.307 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.307 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.307 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.307 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.307 DEBUG:boto:Method: GET 2023-09-29T22:12:36.307 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.307 DEBUG:boto:Data: 2023-09-29T22:12:36.308 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.308 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.308 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.308 DEBUG:boto:Params: {} 2023-09-29T22:12:36.308 DEBUG:boto:Token: None 2023-09-29T22:12:36.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009823f49a60922bc2-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.310 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.310 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.311 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.311 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.311 DEBUG:boto:Method: GET 2023-09-29T22:12:36.311 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.311 DEBUG:boto:Data: 2023-09-29T22:12:36.311 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.311 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.311 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.311 DEBUG:boto:Params: {} 2023-09-29T22:12:36.311 DEBUG:boto:Token: None 2023-09-29T22:12:36.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e4e25be1fe879d6-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.314 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.314 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.314 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.314 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.314 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.314 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.315 DEBUG:boto:Data: 2023-09-29T22:12:36.315 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.315 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.315 DEBUG:boto:Params: {} 2023-09-29T22:12:36.315 DEBUG:boto:Token: None 2023-09-29T22:12:36.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.315 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI= 2023-09-29T22:12:36.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI=', 'Content-Length': '0'} 2023-09-29T22:12:36.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e97caf964be844fd-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.317 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.317 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.317 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.317 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.318 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.318 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.318 DEBUG:boto:Data: 2023-09-29T22:12:36.318 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.318 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.318 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.318 DEBUG:boto:Params: {} 2023-09-29T22:12:36.318 DEBUG:boto:Token: None 2023-09-29T22:12:36.318 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI= 2023-09-29T22:12:36.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI=', 'Content-Length': '0'} 2023-09-29T22:12:36.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c9fdff975d4bfd0-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.321 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.321 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.321 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.321 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.321 DEBUG:boto:Method: GET 2023-09-29T22:12:36.321 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.321 DEBUG:boto:Data: 2023-09-29T22:12:36.321 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.321 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.321 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.322 DEBUG:boto:Params: {} 2023-09-29T22:12:36.322 DEBUG:boto:Token: None 2023-09-29T22:12:36.322 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.322 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030334041a8b2cc2b-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.324 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.324 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.324 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.324 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.324 DEBUG:boto:Method: GET 2023-09-29T22:12:36.324 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.325 DEBUG:boto:Data: 2023-09-29T22:12:36.325 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.325 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.325 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.325 DEBUG:boto:Params: {} 2023-09-29T22:12:36.325 DEBUG:boto:Token: None 2023-09-29T22:12:36.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.325 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002822a99dd6d14c7d-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.327 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.327 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.328 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.328 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.328 DEBUG:boto:Method: GET 2023-09-29T22:12:36.328 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.328 DEBUG:boto:Data: 2023-09-29T22:12:36.328 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.328 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.328 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.328 DEBUG:boto:Params: {} 2023-09-29T22:12:36.328 DEBUG:boto:Token: None 2023-09-29T22:12:36.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.329 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e258bc62d62c484-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.331 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.331 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.331 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.332 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.332 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.332 DEBUG:boto:Method: GET 2023-09-29T22:12:36.332 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.332 DEBUG:boto:Data: 2023-09-29T22:12:36.332 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.332 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.332 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.332 DEBUG:boto:Params: {} 2023-09-29T22:12:36.333 DEBUG:boto:Token: None 2023-09-29T22:12:36.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3387622b2c8927e-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.382 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.383 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.383 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.383 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.383 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.383 DEBUG:boto:Data: 2023-09-29T22:12:36.383 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.384 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.384 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.384 DEBUG:boto:Params: {} 2023-09-29T22:12:36.384 DEBUG:boto:Token: None 2023-09-29T22:12:36.384 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.384 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I= 2023-09-29T22:12:36.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I=', 'Content-Length': '0'} 2023-09-29T22:12:36.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000130346722bff081a-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.386 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.386 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.387 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.387 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.387 DEBUG:boto:Data: 2023-09-29T22:12:36.387 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.387 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.387 DEBUG:boto:Params: {} 2023-09-29T22:12:36.387 DEBUG:boto:Token: None 2023-09-29T22:12:36.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I= 2023-09-29T22:12:36.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I=', 'Content-Length': '0'} 2023-09-29T22:12:36.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d133fed2147085c7-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.390 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.390 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.390 DEBUG:boto:Method: GET 2023-09-29T22:12:36.390 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.390 DEBUG:boto:Data: 2023-09-29T22:12:36.390 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.391 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.391 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.391 DEBUG:boto:Params: {} 2023-09-29T22:12:36.391 DEBUG:boto:Token: None 2023-09-29T22:12:36.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.391 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA= 2023-09-29T22:12:36.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA=', 'Content-Length': '0'} 2023-09-29T22:12:36.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ace21607aaa2f330-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.393 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.393 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.393 DEBUG:boto:Method: GET 2023-09-29T22:12:36.394 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.394 DEBUG:boto:Data: 2023-09-29T22:12:36.394 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.394 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.394 DEBUG:boto:Params: {} 2023-09-29T22:12:36.394 DEBUG:boto:Token: None 2023-09-29T22:12:36.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA= 2023-09-29T22:12:36.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA=', 'Content-Length': '0'} 2023-09-29T22:12:36.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b34b5f1ee579210f-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.396 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.397 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.397 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.397 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.397 DEBUG:boto:Method: GET 2023-09-29T22:12:36.397 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.397 DEBUG:boto:Data: 2023-09-29T22:12:36.397 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.397 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.397 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.397 DEBUG:boto:Params: {} 2023-09-29T22:12:36.398 DEBUG:boto:Token: None 2023-09-29T22:12:36.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.398 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097cc064d4835e226-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.400 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.401 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.401 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.401 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.401 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.401 DEBUG:boto:Method: GET 2023-09-29T22:12:36.401 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.401 DEBUG:boto:Data: 2023-09-29T22:12:36.401 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.401 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.401 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.402 DEBUG:boto:Params: {} 2023-09-29T22:12:36.402 DEBUG:boto:Token: None 2023-09-29T22:12:36.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b090f37c36819824-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.404 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:36.405 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.405 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.405 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.405 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.405 DEBUG:boto:Method: GET 2023-09-29T22:12:36.405 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.406 DEBUG:boto:Data: 2023-09-29T22:12:36.406 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.406 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.406 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.406 DEBUG:boto:Params: {} 2023-09-29T22:12:36.406 DEBUG:boto:Token: None 2023-09-29T22:12:36.406 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.406 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f69f3cacc63c8bd-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.408 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.408 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.409 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.409 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.409 DEBUG:boto:Method: GET 2023-09-29T22:12:36.409 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.409 DEBUG:boto:Data: 2023-09-29T22:12:36.409 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.409 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.409 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.409 DEBUG:boto:Params: {} 2023-09-29T22:12:36.409 DEBUG:boto:Token: None 2023-09-29T22:12:36.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.410 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dee8f76a25f6796c-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.412 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.412 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.412 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.412 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.412 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.412 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.413 DEBUG:boto:Data: 2023-09-29T22:12:36.413 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.413 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.413 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.413 DEBUG:boto:Params: {} 2023-09-29T22:12:36.413 DEBUG:boto:Token: None 2023-09-29T22:12:36.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.413 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ADwzqqivrVud9iCrMqXrRbYtZLo= 2023-09-29T22:12:36.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ADwzqqivrVud9iCrMqXrRbYtZLo=', 'Content-Length': '0'} 2023-09-29T22:12:36.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001413c43e6b230bcf-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.415 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.415 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.415 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.415 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.416 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.416 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.416 DEBUG:boto:Data: 2023-09-29T22:12:36.416 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.416 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.416 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.416 DEBUG:boto:Params: {} 2023-09-29T22:12:36.416 DEBUG:boto:Token: None 2023-09-29T22:12:36.416 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ADwzqqivrVud9iCrMqXrRbYtZLo= 2023-09-29T22:12:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ADwzqqivrVud9iCrMqXrRbYtZLo=', 'Content-Length': '0'} 2023-09-29T22:12:36.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001469a209747409d5-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:36.419 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.419 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.419 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.419 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.419 DEBUG:boto:Method: GET 2023-09-29T22:12:36.419 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.419 DEBUG:boto:Data: 2023-09-29T22:12:36.419 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.419 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.419 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.420 DEBUG:boto:Params: {} 2023-09-29T22:12:36.420 DEBUG:boto:Token: None 2023-09-29T22:12:36.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abbe9f27b08df545-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.466 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.467 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.467 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.467 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.467 DEBUG:boto:Method: GET 2023-09-29T22:12:36.467 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.467 DEBUG:boto:Data: 2023-09-29T22:12:36.467 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.467 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.467 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.468 DEBUG:boto:Params: {} 2023-09-29T22:12:36.468 DEBUG:boto:Token: None 2023-09-29T22:12:36.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.468 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016ef1de3fd6aa914-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.470 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.470 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.471 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.471 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.471 DEBUG:boto:Method: GET 2023-09-29T22:12:36.471 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.471 DEBUG:boto:Data: 2023-09-29T22:12:36.471 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.471 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.471 DEBUG:boto:Params: {} 2023-09-29T22:12:36.471 DEBUG:boto:Token: None 2023-09-29T22:12:36.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8e31e9cca544d48-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.474 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.474 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.475 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.475 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.475 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.475 DEBUG:boto:Method: GET 2023-09-29T22:12:36.475 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.475 DEBUG:boto:Data: 2023-09-29T22:12:36.475 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.475 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.475 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.476 DEBUG:boto:Params: {} 2023-09-29T22:12:36.476 DEBUG:boto:Token: None 2023-09-29T22:12:36.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e6468548aa8488-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.478 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.479 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.479 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.479 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.479 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:36.479 DEBUG:boto:Data: 2023-09-29T22:12:36.479 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.479 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.480 DEBUG:boto:Params: {} 2023-09-29T22:12:36.480 DEBUG:boto:Token: None 2023-09-29T22:12:36.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:36.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e1XFLSWuq4XLKytJR8Oo2j77j3M= 2023-09-29T22:12:36.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e1XFLSWuq4XLKytJR8Oo2j77j3M=', 'Content-Length': '0'} 2023-09-29T22:12:36.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2198b87edcc65b8-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.482 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.482 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.482 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.482 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:36.482 DEBUG:boto:Data: 2023-09-29T22:12:36.482 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.483 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.483 DEBUG:boto:Params: {} 2023-09-29T22:12:36.483 DEBUG:boto:Token: None 2023-09-29T22:12:36.483 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:36.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e1XFLSWuq4XLKytJR8Oo2j77j3M= 2023-09-29T22:12:36.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e1XFLSWuq4XLKytJR8Oo2j77j3M=', 'Content-Length': '0'} 2023-09-29T22:12:36.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034c884b1ddde2ae4-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:36.485 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.485 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.486 DEBUG:boto:Method: GET 2023-09-29T22:12:36.486 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:36.486 DEBUG:boto:Data: 2023-09-29T22:12:36.486 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.486 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.486 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.486 DEBUG:boto:Params: {} 2023-09-29T22:12:36.486 DEBUG:boto:Token: None 2023-09-29T22:12:36.486 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:36.487 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R4E5Pd0u/Ba8dDb7mQRLOJTi6kQ= 2023-09-29T22:12:36.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R4E5Pd0u/Ba8dDb7mQRLOJTi6kQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000369cd5238a678213-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.488 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.489 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.489 DEBUG:boto:Method: GET 2023-09-29T22:12:36.489 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:36.489 DEBUG:boto:Data: 2023-09-29T22:12:36.489 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.489 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.489 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.489 DEBUG:boto:Params: {} 2023-09-29T22:12:36.489 DEBUG:boto:Token: None 2023-09-29T22:12:36.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:36.490 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R4E5Pd0u/Ba8dDb7mQRLOJTi6kQ= 2023-09-29T22:12:36.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R4E5Pd0u/Ba8dDb7mQRLOJTi6kQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc93f316b82cdedd-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.492 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.492 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.492 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.492 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.492 DEBUG:boto:Method: GET 2023-09-29T22:12:36.492 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.492 DEBUG:boto:Data: 2023-09-29T22:12:36.492 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.492 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.492 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.493 DEBUG:boto:Params: {} 2023-09-29T22:12:36.493 DEBUG:boto:Token: None 2023-09-29T22:12:36.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.493 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c0d739a1e2f8e2-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.495 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.496 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.496 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.496 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.496 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.496 DEBUG:boto:Method: GET 2023-09-29T22:12:36.496 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:36.496 DEBUG:boto:Data: 2023-09-29T22:12:36.496 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.497 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.497 DEBUG:boto:Params: {} 2023-09-29T22:12:36.497 DEBUG:boto:Token: None 2023-09-29T22:12:36.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.497 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000889991a711eee6ee-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.499 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:36.500 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.500 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.500 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.500 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.500 DEBUG:boto:Method: GET 2023-09-29T22:12:36.500 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.501 DEBUG:boto:Data: 2023-09-29T22:12:36.501 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.501 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.501 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.501 DEBUG:boto:Params: {} 2023-09-29T22:12:36.501 DEBUG:boto:Token: None 2023-09-29T22:12:36.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.501 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo= 2023-09-29T22:12:36.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo=', 'Content-Length': '0'} 2023-09-29T22:12:36.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092073762e7700cf2-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.503 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.503 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.503 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.503 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.504 DEBUG:boto:Method: GET 2023-09-29T22:12:36.504 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.504 DEBUG:boto:Data: 2023-09-29T22:12:36.504 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.504 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.504 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.504 DEBUG:boto:Params: {} 2023-09-29T22:12:36.504 DEBUG:boto:Token: None 2023-09-29T22:12:36.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo= 2023-09-29T22:12:36.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo=', 'Content-Length': '0'} 2023-09-29T22:12:36.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dddcddac274acd6d-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.550 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.551 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.551 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.551 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.551 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.551 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.551 DEBUG:boto:Data: 2023-09-29T22:12:36.551 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.551 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.551 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.552 DEBUG:boto:Params: {} 2023-09-29T22:12:36.552 DEBUG:boto:Token: None 2023-09-29T22:12:36.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cizGAVWUj8r12ix4YP0+17usCmQ= 2023-09-29T22:12:36.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cizGAVWUj8r12ix4YP0+17usCmQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afcacbab96da6394-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.554 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.554 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.555 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.555 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.555 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.555 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.555 DEBUG:boto:Data: 2023-09-29T22:12:36.555 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.555 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.555 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.555 DEBUG:boto:Params: {} 2023-09-29T22:12:36.555 DEBUG:boto:Token: None 2023-09-29T22:12:36.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.556 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cizGAVWUj8r12ix4YP0+17usCmQ= 2023-09-29T22:12:36.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cizGAVWUj8r12ix4YP0+17usCmQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002203aa5bb53cb722-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:36.558 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.558 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.558 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.558 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.558 DEBUG:boto:Method: GET 2023-09-29T22:12:36.559 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.559 DEBUG:boto:Data: 2023-09-29T22:12:36.559 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.559 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.559 DEBUG:boto:Params: {} 2023-09-29T22:12:36.559 DEBUG:boto:Token: None 2023-09-29T22:12:36.559 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.559 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo= 2023-09-29T22:12:36.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo=', 'Content-Length': '0'} 2023-09-29T22:12:36.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc08574edf1a1c05-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.561 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.561 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.562 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.562 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.562 DEBUG:boto:Method: GET 2023-09-29T22:12:36.562 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.562 DEBUG:boto:Data: 2023-09-29T22:12:36.562 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.562 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.562 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.562 DEBUG:boto:Params: {} 2023-09-29T22:12:36.562 DEBUG:boto:Token: None 2023-09-29T22:12:36.563 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:36.563 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo= 2023-09-29T22:12:36.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5MsZjoM1SJO6DJNT/MW2J43GiWo=', 'Content-Length': '0'} 2023-09-29T22:12:36.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a00848787f86475-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.565 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.565 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.565 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.565 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.565 DEBUG:boto:Method: GET 2023-09-29T22:12:36.565 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.565 DEBUG:boto:Data: 2023-09-29T22:12:36.566 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.566 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.566 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.566 DEBUG:boto:Params: {} 2023-09-29T22:12:36.566 DEBUG:boto:Token: None 2023-09-29T22:12:36.566 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.566 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000720265e82edbbc6-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.569 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.569 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.569 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.569 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.570 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.570 DEBUG:boto:Method: GET 2023-09-29T22:12:36.570 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.570 DEBUG:boto:Data: 2023-09-29T22:12:36.570 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.570 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.570 DEBUG:boto:Params: {} 2023-09-29T22:12:36.570 DEBUG:boto:Token: None 2023-09-29T22:12:36.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48df886f73a4620-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.573 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.573 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.574 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.574 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.574 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:36.574 DEBUG:boto:Data: 2023-09-29T22:12:36.574 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.574 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.574 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.574 DEBUG:boto:Params: {} 2023-09-29T22:12:36.574 DEBUG:boto:Token: None 2023-09-29T22:12:36.575 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj 2023-09-29T22:12:36.575 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TXp/ovrcI/+epvwm9x0hVQ5xbwc= 2023-09-29T22:12:36.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TXp/ovrcI/+epvwm9x0hVQ5xbwc=', 'Content-Length': '0'} 2023-09-29T22:12:36.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000316ec6fe79da7711-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.576 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.576 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.576 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.577 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:36.577 DEBUG:boto:Data: 2023-09-29T22:12:36.577 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.577 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.577 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.577 DEBUG:boto:Params: {} 2023-09-29T22:12:36.577 DEBUG:boto:Token: None 2023-09-29T22:12:36.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj 2023-09-29T22:12:36.577 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TXp/ovrcI/+epvwm9x0hVQ5xbwc= 2023-09-29T22:12:36.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TXp/ovrcI/+epvwm9x0hVQ5xbwc=', 'Content-Length': '0'} 2023-09-29T22:12:36.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1bca114a6f7750c-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:36.579 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.579 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.580 DEBUG:boto:Method: GET 2023-09-29T22:12:36.580 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:36.580 DEBUG:boto:Data: 2023-09-29T22:12:36.580 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.580 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.580 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.580 DEBUG:boto:Params: {} 2023-09-29T22:12:36.580 DEBUG:boto:Token: None 2023-09-29T22:12:36.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj 2023-09-29T22:12:36.581 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6ZNw8Tk6QpnHFh2OxsrpkRdlxmc= 2023-09-29T22:12:36.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6ZNw8Tk6QpnHFh2OxsrpkRdlxmc=', 'Content-Length': '0'} 2023-09-29T22:12:36.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063f5e6c0ba2ae3eb-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.582 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:36.582 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:36.582 DEBUG:boto:Method: GET 2023-09-29T22:12:36.583 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:36.583 DEBUG:boto:Data: 2023-09-29T22:12:36.583 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.583 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.583 DEBUG:boto:Params: {} 2023-09-29T22:12:36.583 DEBUG:boto:Token: None 2023-09-29T22:12:36.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/myobj 2023-09-29T22:12:36.583 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6ZNw8Tk6QpnHFh2OxsrpkRdlxmc= 2023-09-29T22:12:36.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6ZNw8Tk6QpnHFh2OxsrpkRdlxmc=', 'Content-Length': '0'} 2023-09-29T22:12:36.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000273aee5aff0222d7-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.585 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.586 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.586 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.586 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.586 DEBUG:boto:Method: GET 2023-09-29T22:12:36.586 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.586 DEBUG:boto:Data: 2023-09-29T22:12:36.586 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.586 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:36.586 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.587 DEBUG:boto:Params: {} 2023-09-29T22:12:36.587 DEBUG:boto:Token: None 2023-09-29T22:12:36.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.587 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092863aa98ad41355-0065174bd4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.589 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.590 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.590 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.590 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.590 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.590 DEBUG:boto:Method: GET 2023-09-29T22:12:36.590 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:36.590 DEBUG:boto:Data: 2023-09-29T22:12:36.591 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.591 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.591 DEBUG:boto:Params: {} 2023-09-29T22:12:36.591 DEBUG:boto:Token: None 2023-09-29T22:12:36.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005241c05dec731024-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.594 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-12 zones={test-zone1, test-zone3} 2023-09-29T22:12:36.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:36.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:36.594 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:36.655 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:36.692 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:36.691+0000 7f3488215880 1 bucket sync caught up with source: 2023-09-29T22:12:36.692 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.6, , 00000000001.18.6, , , , 00000000001.27.6, 00000000001.11.6, , ] 2023-09-29T22:12:36.692 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.10.6, , 00000000001.18.6, , , , 00000000001.27.6, 00000000001.11.6, , ] 2023-09-29T22:12:36.692 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:36.691+0000 7f3488215880 0 bucket checkpoint complete 2023-09-29T22:12:36.704 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-12 zones={test-zone2, test-zone3} 2023-09-29T22:12:36.704 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.705 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.705 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.705 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:36.705 DEBUG:boto:Data: 2023-09-29T22:12:36.705 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.705 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.705 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.705 DEBUG:boto:Params: {} 2023-09-29T22:12:36.706 DEBUG:boto:Token: None 2023-09-29T22:12:36.706 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.706 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78= 2023-09-29T22:12:36.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78=', 'Content-Length': '0'} 2023-09-29T22:12:36.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4b332f3cee3119b-0065174bd4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.709 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.709 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.709 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.709 DEBUG:boto:Path: /dmuhsu-12/ 2023-09-29T22:12:36.709 DEBUG:boto:Data: 2023-09-29T22:12:36.709 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.709 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.710 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.710 DEBUG:boto:Params: {} 2023-09-29T22:12:36.710 DEBUG:boto:Token: None 2023-09-29T22:12:36.710 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.710 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78= 2023-09-29T22:12:36.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oe4WdXJAyLjN+SaknL1WV/ERJ78=', 'Content-Length': '0'} 2023-09-29T22:12:36.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fad1191b804e27db-0065174bd4-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:12:36.713 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.713 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.713 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.713 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.713 DEBUG:boto:Method: GET 2023-09-29T22:12:36.713 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.713 DEBUG:boto:Data: 2023-09-29T22:12:36.713 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.714 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.714 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.714 DEBUG:boto:Params: {} 2023-09-29T22:12:36.714 DEBUG:boto:Token: None 2023-09-29T22:12:36.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.714 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018df8be51534fa5b-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.716 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:36.717 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:36.717 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:12:36.718 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.718 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.718 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.718 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.718 DEBUG:boto:Method: GET 2023-09-29T22:12:36.718 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.718 DEBUG:boto:Data: 2023-09-29T22:12:36.718 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.719 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.719 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.719 DEBUG:boto:Params: {} 2023-09-29T22:12:36.719 DEBUG:boto:Token: None 2023-09-29T22:12:36.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002533cbfd425c1796-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.721 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.722 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T22:12:36.722 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T22:12:36.722 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T22:12:36.722 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:12:36.722 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.723 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.723 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.723 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.723 DEBUG:boto:Method: GET 2023-09-29T22:12:36.723 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.723 DEBUG:boto:Data: 2023-09-29T22:12:36.723 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.723 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.723 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.723 DEBUG:boto:Params: {} 2023-09-29T22:12:36.724 DEBUG:boto:Token: None 2023-09-29T22:12:36.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f63a33c37553cfb6-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.726 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.727 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.727 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.727 DEBUG:boto:path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.727 DEBUG:boto:auth_path=/dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.727 DEBUG:boto:Method: GET 2023-09-29T22:12:36.727 DEBUG:boto:Path: /dmuhsu-12/?versions&max-keys=999 2023-09-29T22:12:36.727 DEBUG:boto:Data: 2023-09-29T22:12:36.727 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.727 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.727 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.728 DEBUG:boto:Params: {} 2023-09-29T22:12:36.728 DEBUG:boto:Token: None 2023-09-29T22:12:36.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/?versions 2023-09-29T22:12:36.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ= 2023-09-29T22:12:36.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/f4iMol+Mgds1Aq7PPIx7sEm4yQ=', 'Content-Length': '0'} 2023-09-29T22:12:36.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043289684aec29b48-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.730 DEBUG:boto:b'dmuhsu-12999false&nulltrue2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.731 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.731 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.731 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.731 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.731 DEBUG:boto:Method: GET 2023-09-29T22:12:36.732 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.732 DEBUG:boto:Data: 2023-09-29T22:12:36.732 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.732 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.732 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.732 DEBUG:boto:Params: {} 2023-09-29T22:12:36.732 DEBUG:boto:Token: None 2023-09-29T22:12:36.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ffdfb80971ecd4a-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.734 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.734 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.735 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.735 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.735 DEBUG:boto:Method: GET 2023-09-29T22:12:36.735 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.735 DEBUG:boto:Data: 2023-09-29T22:12:36.735 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.735 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.735 DEBUG:boto:Params: {} 2023-09-29T22:12:36.735 DEBUG:boto:Token: None 2023-09-29T22:12:36.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040439ac92bc0e19d-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.778 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.779 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.779 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.779 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.779 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.779 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.779 DEBUG:boto:Data: 2023-09-29T22:12:36.779 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.779 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.779 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.780 DEBUG:boto:Params: {} 2023-09-29T22:12:36.780 DEBUG:boto:Token: None 2023-09-29T22:12:36.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM= 2023-09-29T22:12:36.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM=', 'Content-Length': '0'} 2023-09-29T22:12:36.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d17f4db7b394c4eb-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.782 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.782 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.782 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.782 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.782 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.782 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.783 DEBUG:boto:Data: 2023-09-29T22:12:36.783 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.783 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.783 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.783 DEBUG:boto:Params: {} 2023-09-29T22:12:36.783 DEBUG:boto:Token: None 2023-09-29T22:12:36.783 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.783 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM= 2023-09-29T22:12:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ICMPNaJYYjPK9bGi0C18M+/rvzM=', 'Content-Length': '0'} 2023-09-29T22:12:36.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6ab30fa878fb42e-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.785 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.786 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.786 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.786 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.786 DEBUG:boto:Method: GET 2023-09-29T22:12:36.786 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.786 DEBUG:boto:Data: 2023-09-29T22:12:36.786 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.786 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.786 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.786 DEBUG:boto:Params: {} 2023-09-29T22:12:36.787 DEBUG:boto:Token: None 2023-09-29T22:12:36.787 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.787 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0a585eca3636dd5-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.789 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.789 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.789 DEBUG:boto:path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.789 DEBUG:boto:auth_path=/dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.789 DEBUG:boto:Method: GET 2023-09-29T22:12:36.789 DEBUG:boto:Path: /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.790 DEBUG:boto:Data: 2023-09-29T22:12:36.790 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.790 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.790 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.790 DEBUG:boto:Params: {} 2023-09-29T22:12:36.790 DEBUG:boto:Token: None 2023-09-29T22:12:36.790 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26?versionId=null 2023-09-29T22:12:36.790 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0= 2023-09-29T22:12:36.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ePpP4SGlfiYYUqAU/lkXiSQGsb0=', 'Content-Length': '0'} 2023-09-29T22:12:36.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e44a561e07b6236-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.792 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.792 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.793 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.793 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.793 DEBUG:boto:Method: GET 2023-09-29T22:12:36.793 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.793 DEBUG:boto:Data: 2023-09-29T22:12:36.793 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.793 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.793 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.793 DEBUG:boto:Params: {} 2023-09-29T22:12:36.793 DEBUG:boto:Token: None 2023-09-29T22:12:36.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.794 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bdfc7ae71df585d-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.796 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.797 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.797 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.797 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.797 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.797 DEBUG:boto:Method: GET 2023-09-29T22:12:36.797 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.797 DEBUG:boto:Data: 2023-09-29T22:12:36.797 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.797 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.798 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.798 DEBUG:boto:Params: {} 2023-09-29T22:12:36.798 DEBUG:boto:Token: None 2023-09-29T22:12:36.798 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.798 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596183203c4aeb2c-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.801 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.801 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.801 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.801 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.801 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.801 DEBUG:boto:Data: 2023-09-29T22:12:36.802 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.802 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.802 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.802 DEBUG:boto:Params: {} 2023-09-29T22:12:36.802 DEBUG:boto:Token: None 2023-09-29T22:12:36.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY= 2023-09-29T22:12:36.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY=', 'Content-Length': '0'} 2023-09-29T22:12:36.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b5187107dd3754-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.804 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.804 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.804 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.805 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.805 DEBUG:boto:Data: 2023-09-29T22:12:36.805 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.805 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.805 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.805 DEBUG:boto:Params: {} 2023-09-29T22:12:36.805 DEBUG:boto:Token: None 2023-09-29T22:12:36.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY= 2023-09-29T22:12:36.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lh0+3ix79dYCb+O7QvEFBEYsBdY=', 'Content-Length': '0'} 2023-09-29T22:12:36.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cda7ab433f1e2dcd-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T22:12:36.808 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.808 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.808 DEBUG:boto:Method: GET 2023-09-29T22:12:36.808 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.808 DEBUG:boto:Data: 2023-09-29T22:12:36.808 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.808 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.808 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.808 DEBUG:boto:Params: {} 2023-09-29T22:12:36.808 DEBUG:boto:Token: None 2023-09-29T22:12:36.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE= 2023-09-29T22:12:36.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE=', 'Content-Length': '0'} 2023-09-29T22:12:36.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081851e180ee34b55-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.811 DEBUG:boto:path=/dmuhsu-12/%26 2023-09-29T22:12:36.811 DEBUG:boto:auth_path=/dmuhsu-12/%26 2023-09-29T22:12:36.811 DEBUG:boto:Method: GET 2023-09-29T22:12:36.811 DEBUG:boto:Path: /dmuhsu-12/%26 2023-09-29T22:12:36.811 DEBUG:boto:Data: 2023-09-29T22:12:36.811 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.811 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.811 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.811 DEBUG:boto:Params: {} 2023-09-29T22:12:36.812 DEBUG:boto:Token: None 2023-09-29T22:12:36.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%26 2023-09-29T22:12:36.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE= 2023-09-29T22:12:36.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1CtkmScAiU04dW8vb6sFZS4sSeE=', 'Content-Length': '0'} 2023-09-29T22:12:36.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093cbff0c9c7d8241-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.814 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.814 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.814 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.814 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.815 DEBUG:boto:Method: GET 2023-09-29T22:12:36.815 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.815 DEBUG:boto:Data: 2023-09-29T22:12:36.815 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.815 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.815 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.815 DEBUG:boto:Params: {} 2023-09-29T22:12:36.815 DEBUG:boto:Token: None 2023-09-29T22:12:36.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.815 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880c42681988c127-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.862 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.863 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.863 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.863 DEBUG:boto:path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.863 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.863 DEBUG:boto:Method: GET 2023-09-29T22:12:36.863 DEBUG:boto:Path: /dmuhsu-12/?prefix=%26 2023-09-29T22:12:36.864 DEBUG:boto:Data: 2023-09-29T22:12:36.864 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.864 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.864 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.864 DEBUG:boto:Params: {} 2023-09-29T22:12:36.864 DEBUG:boto:Token: None 2023-09-29T22:12:36.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d695afc2203880ac-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.867 DEBUG:boto:b'dmuhsu-12&1000false&2023-09-29T22:12:07.959Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.868 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.868 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.868 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.868 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.869 DEBUG:boto:Method: GET 2023-09-29T22:12:36.869 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.869 DEBUG:boto:Data: 2023-09-29T22:12:36.869 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.869 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.869 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.869 DEBUG:boto:Params: {} 2023-09-29T22:12:36.869 DEBUG:boto:Token: None 2023-09-29T22:12:36.870 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a53c58cba50077a-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.872 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.872 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.873 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.873 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.873 DEBUG:boto:Method: GET 2023-09-29T22:12:36.873 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.873 DEBUG:boto:Data: 2023-09-29T22:12:36.873 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.873 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.873 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.873 DEBUG:boto:Params: {} 2023-09-29T22:12:36.874 DEBUG:boto:Token: None 2023-09-29T22:12:36.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.874 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075dd2bd93f60ba83-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.918 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.919 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.919 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.919 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.919 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.919 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.919 DEBUG:boto:Data: 2023-09-29T22:12:36.919 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.919 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.919 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.920 DEBUG:boto:Params: {} 2023-09-29T22:12:36.920 DEBUG:boto:Token: None 2023-09-29T22:12:36.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI= 2023-09-29T22:12:36.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI=', 'Content-Length': '0'} 2023-09-29T22:12:36.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000086e9716074838a8-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.922 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.922 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.922 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.923 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.923 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.923 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.923 DEBUG:boto:Data: 2023-09-29T22:12:36.923 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.923 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.923 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.923 DEBUG:boto:Params: {} 2023-09-29T22:12:36.923 DEBUG:boto:Token: None 2023-09-29T22:12:36.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI= 2023-09-29T22:12:36.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9sgi0SGLVcrn9Q9zBtnKI+A7QXI=', 'Content-Length': '0'} 2023-09-29T22:12:36.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066e6b6edce4e1af1-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.926 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.926 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.926 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.926 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.927 DEBUG:boto:Method: GET 2023-09-29T22:12:36.927 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.927 DEBUG:boto:Data: 2023-09-29T22:12:36.927 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.927 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.927 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.927 DEBUG:boto:Params: {} 2023-09-29T22:12:36.927 DEBUG:boto:Token: None 2023-09-29T22:12:36.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002718fc3d164dd2a-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.930 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.930 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.930 DEBUG:boto:path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.930 DEBUG:boto:auth_path=/dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.930 DEBUG:boto:Method: GET 2023-09-29T22:12:36.931 DEBUG:boto:Path: /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.931 DEBUG:boto:Data: 2023-09-29T22:12:36.931 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.931 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.931 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.931 DEBUG:boto:Params: {} 2023-09-29T22:12:36.931 DEBUG:boto:Token: None 2023-09-29T22:12:36.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A?versionId=null 2023-09-29T22:12:36.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU= 2023-09-29T22:12:36.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mCs0XikTtBmbkx5ELv2R8mPegeU=', 'Content-Length': '0'} 2023-09-29T22:12:36.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091c0c0b94459032b-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.933 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.934 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.934 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.934 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.934 DEBUG:boto:Method: GET 2023-09-29T22:12:36.934 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.934 DEBUG:boto:Data: 2023-09-29T22:12:36.934 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.934 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.935 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.935 DEBUG:boto:Params: {} 2023-09-29T22:12:36.935 DEBUG:boto:Token: None 2023-09-29T22:12:36.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.935 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c4e97718662a0f-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.938 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.938 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.938 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.938 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.938 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.939 DEBUG:boto:Method: GET 2023-09-29T22:12:36.939 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.939 DEBUG:boto:Data: 2023-09-29T22:12:36.939 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.939 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.939 DEBUG:boto:Params: {} 2023-09-29T22:12:36.939 DEBUG:boto:Token: None 2023-09-29T22:12:36.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8d197df2d500fdf-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.943 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.943 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.943 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.943 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.943 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.944 DEBUG:boto:Data: 2023-09-29T22:12:36.944 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.944 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.944 DEBUG:boto:Params: {} 2023-09-29T22:12:36.944 DEBUG:boto:Token: None 2023-09-29T22:12:36.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I= 2023-09-29T22:12:36.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I=', 'Content-Length': '0'} 2023-09-29T22:12:36.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbbddcf5ecf79577-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.946 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.947 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.947 DEBUG:boto:Method: HEAD 2023-09-29T22:12:36.947 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.947 DEBUG:boto:Data: 2023-09-29T22:12:36.947 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.947 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.947 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.947 DEBUG:boto:Params: {} 2023-09-29T22:12:36.948 DEBUG:boto:Token: None 2023-09-29T22:12:36.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I= 2023-09-29T22:12:36.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8gdo8xWTOKYcnQ4lUNjY2A+bL4I=', 'Content-Length': '0'} 2023-09-29T22:12:36.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff188702ca3167aa-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T22:12:36.950 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.950 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.951 DEBUG:boto:Method: GET 2023-09-29T22:12:36.951 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.951 DEBUG:boto:Data: 2023-09-29T22:12:36.951 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.951 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.951 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.951 DEBUG:boto:Params: {} 2023-09-29T22:12:36.952 DEBUG:boto:Token: None 2023-09-29T22:12:36.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA= 2023-09-29T22:12:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA=', 'Content-Length': '0'} 2023-09-29T22:12:36.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008554a39ada31ae76-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.954 DEBUG:boto:path=/dmuhsu-12/%3A 2023-09-29T22:12:36.954 DEBUG:boto:auth_path=/dmuhsu-12/%3A 2023-09-29T22:12:36.954 DEBUG:boto:Method: GET 2023-09-29T22:12:36.954 DEBUG:boto:Path: /dmuhsu-12/%3A 2023-09-29T22:12:36.955 DEBUG:boto:Data: 2023-09-29T22:12:36.955 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.955 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.955 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.955 DEBUG:boto:Params: {} 2023-09-29T22:12:36.955 DEBUG:boto:Token: None 2023-09-29T22:12:36.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/%3A 2023-09-29T22:12:36.955 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA= 2023-09-29T22:12:36.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RMcbdbYC9v5DVaS4R3zgTk9BMtA=', 'Content-Length': '0'} 2023-09-29T22:12:36.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d35328a1a3d7cc6f-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.958 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.958 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.958 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.958 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.958 DEBUG:boto:Method: GET 2023-09-29T22:12:36.958 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.958 DEBUG:boto:Data: 2023-09-29T22:12:36.959 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.959 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.959 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.959 DEBUG:boto:Params: {} 2023-09-29T22:12:36.959 DEBUG:boto:Token: None 2023-09-29T22:12:36.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008853ed445aaeb908-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.962 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.962 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:36.963 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:36.963 DEBUG:boto:path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.963 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.963 DEBUG:boto:Method: GET 2023-09-29T22:12:36.963 DEBUG:boto:Path: /dmuhsu-12/?prefix=%3A 2023-09-29T22:12:36.963 DEBUG:boto:Data: 2023-09-29T22:12:36.963 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.964 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.964 DEBUG:boto:Params: {} 2023-09-29T22:12:36.964 DEBUG:boto:Token: None 2023-09-29T22:12:36.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/ 2023-09-29T22:12:36.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s= 2023-09-29T22:12:36.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gSGJqsV1cqW6ZiidhJ4afDKMV9s=', 'Content-Length': '0'} 2023-09-29T22:12:36.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002362d703ce99991a-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.967 DEBUG:boto:b'dmuhsu-12:1000false:2023-09-29T22:12:07.910Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:36.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:36.967 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.968 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.968 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.968 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.968 DEBUG:boto:Method: GET 2023-09-29T22:12:36.968 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.968 DEBUG:boto:Data: 2023-09-29T22:12:36.968 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.969 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:36.969 DEBUG:boto:Port: 8000 2023-09-29T22:12:36.969 DEBUG:boto:Params: {} 2023-09-29T22:12:36.969 DEBUG:boto:Token: None 2023-09-29T22:12:36.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000631603b7e845a76b-0065174bd4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:36.971 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.971 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:36.971 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.972 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.972 DEBUG:boto:Method: GET 2023-09-29T22:12:36.972 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.972 DEBUG:boto:Data: 2023-09-29T22:12:36.972 DEBUG:boto:Headers: {} 2023-09-29T22:12:36.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:36.972 DEBUG:boto:Port: 8001 2023-09-29T22:12:36.972 DEBUG:boto:Params: {} 2023-09-29T22:12:36.973 DEBUG:boto:Token: None 2023-09-29T22:12:36.973 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:36 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:36.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU= 2023-09-29T22:12:36.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipMh8tOKCXSOcle/RHUdvtMB0IU=', 'Content-Length': '0'} 2023-09-29T22:12:36.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb4dc4568ff66587-0065174bd4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.022 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.023 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.023 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.023 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.023 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.023 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.023 DEBUG:boto:Data: 2023-09-29T22:12:37.023 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.023 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.023 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.023 DEBUG:boto:Params: {} 2023-09-29T22:12:37.024 DEBUG:boto:Token: None 2023-09-29T22:12:37.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+2V9QmmKko2eKVIlPhozumB7By4= 2023-09-29T22:12:37.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+2V9QmmKko2eKVIlPhozumB7By4=', 'Content-Length': '0'} 2023-09-29T22:12:37.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcbe72ed83df8ff1-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.026 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.027 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.027 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.027 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.027 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.027 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.027 DEBUG:boto:Data: 2023-09-29T22:12:37.027 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.027 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.028 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.028 DEBUG:boto:Params: {} 2023-09-29T22:12:37.028 DEBUG:boto:Token: None 2023-09-29T22:12:37.029 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+2V9QmmKko2eKVIlPhozumB7By4= 2023-09-29T22:12:37.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+2V9QmmKko2eKVIlPhozumB7By4=', 'Content-Length': '0'} 2023-09-29T22:12:37.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031ae61fb988cbb74-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:37.031 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.031 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.031 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.032 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.032 DEBUG:boto:Method: GET 2023-09-29T22:12:37.032 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.032 DEBUG:boto:Data: 2023-09-29T22:12:37.032 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.032 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.032 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.032 DEBUG:boto:Params: {} 2023-09-29T22:12:37.032 DEBUG:boto:Token: None 2023-09-29T22:12:37.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.033 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rwyD+gFP7bwGXCPmlaE0/TDCenc= 2023-09-29T22:12:37.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rwyD+gFP7bwGXCPmlaE0/TDCenc=', 'Content-Length': '0'} 2023-09-29T22:12:37.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f2a3acd1e4c8985-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.035 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.035 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.035 DEBUG:boto:path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.035 DEBUG:boto:auth_path=/dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.035 DEBUG:boto:Method: GET 2023-09-29T22:12:37.035 DEBUG:boto:Path: /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.037 DEBUG:boto:Data: 2023-09-29T22:12:37.037 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.037 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.037 DEBUG:boto:Params: {} 2023-09-29T22:12:37.037 DEBUG:boto:Token: None 2023-09-29T22:12:37.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj?versionId=null 2023-09-29T22:12:37.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rwyD+gFP7bwGXCPmlaE0/TDCenc= 2023-09-29T22:12:37.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rwyD+gFP7bwGXCPmlaE0/TDCenc=', 'Content-Length': '0'} 2023-09-29T22:12:37.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cccfaa877867676-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.040 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.041 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.041 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.041 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.041 DEBUG:boto:Method: GET 2023-09-29T22:12:37.041 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.041 DEBUG:boto:Data: 2023-09-29T22:12:37.041 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.041 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.041 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.042 DEBUG:boto:Params: {} 2023-09-29T22:12:37.042 DEBUG:boto:Token: None 2023-09-29T22:12:37.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.042 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004dc9e1dccaf13a9-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.044 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.045 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.045 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.045 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.045 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.045 DEBUG:boto:Method: GET 2023-09-29T22:12:37.046 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.046 DEBUG:boto:Data: 2023-09-29T22:12:37.046 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.046 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.046 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.046 DEBUG:boto:Params: {} 2023-09-29T22:12:37.046 DEBUG:boto:Token: None 2023-09-29T22:12:37.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3ee3b51cb3a8f65-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.049 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.050 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.050 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.050 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.050 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:37.050 DEBUG:boto:Data: 2023-09-29T22:12:37.050 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.050 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.050 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.051 DEBUG:boto:Params: {} 2023-09-29T22:12:37.051 DEBUG:boto:Token: None 2023-09-29T22:12:37.051 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:37.051 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I3/a+OnS0G4u/2xSAXhfsjUtJNk= 2023-09-29T22:12:37.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I3/a+OnS0G4u/2xSAXhfsjUtJNk=', 'Content-Length': '0'} 2023-09-29T22:12:37.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a16d03cff278fb3f-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.053 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.053 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.053 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.053 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:37.053 DEBUG:boto:Data: 2023-09-29T22:12:37.053 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.053 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.054 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.054 DEBUG:boto:Params: {} 2023-09-29T22:12:37.054 DEBUG:boto:Token: None 2023-09-29T22:12:37.054 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:37.054 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I3/a+OnS0G4u/2xSAXhfsjUtJNk= 2023-09-29T22:12:37.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I3/a+OnS0G4u/2xSAXhfsjUtJNk=', 'Content-Length': '0'} 2023-09-29T22:12:37.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090ba4716a83b6fff-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T22:12:37.056 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.056 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.056 DEBUG:boto:Method: GET 2023-09-29T22:12:37.056 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:37.057 DEBUG:boto:Data: 2023-09-29T22:12:37.057 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.057 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.057 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.057 DEBUG:boto:Params: {} 2023-09-29T22:12:37.057 DEBUG:boto:Token: None 2023-09-29T22:12:37.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:37.057 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6g1Rh2al82FrQRmS2x4aWIFhmsE= 2023-09-29T22:12:37.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6g1Rh2al82FrQRmS2x4aWIFhmsE=', 'Content-Length': '0'} 2023-09-29T22:12:37.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d289bdfcd80cca94-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.059 DEBUG:boto:path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.060 DEBUG:boto:auth_path=/dmuhsu-12/_myobj 2023-09-29T22:12:37.060 DEBUG:boto:Method: GET 2023-09-29T22:12:37.060 DEBUG:boto:Path: /dmuhsu-12/_myobj 2023-09-29T22:12:37.060 DEBUG:boto:Data: 2023-09-29T22:12:37.060 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.060 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.060 DEBUG:boto:Params: {} 2023-09-29T22:12:37.060 DEBUG:boto:Token: None 2023-09-29T22:12:37.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/_myobj 2023-09-29T22:12:37.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6g1Rh2al82FrQRmS2x4aWIFhmsE= 2023-09-29T22:12:37.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6g1Rh2al82FrQRmS2x4aWIFhmsE=', 'Content-Length': '0'} 2023-09-29T22:12:37.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010669fd94176d3d8-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.063 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.063 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.063 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.063 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.063 DEBUG:boto:Method: GET 2023-09-29T22:12:37.063 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.063 DEBUG:boto:Data: 2023-09-29T22:12:37.063 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.064 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.064 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.064 DEBUG:boto:Params: {} 2023-09-29T22:12:37.064 DEBUG:boto:Token: None 2023-09-29T22:12:37.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9273d6499e5890-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.110 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.111 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.111 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.111 DEBUG:boto:path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.111 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.112 DEBUG:boto:Method: GET 2023-09-29T22:12:37.112 DEBUG:boto:Path: /dmuhsu-12/?prefix=_myobj 2023-09-29T22:12:37.112 DEBUG:boto:Data: 2023-09-29T22:12:37.112 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.112 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.112 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.112 DEBUG:boto:Params: {} 2023-09-29T22:12:37.112 DEBUG:boto:Token: None 2023-09-29T22:12:37.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.113 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9dcf5ac2b98c324-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.116 DEBUG:boto:b'dmuhsu-12_myobj1000false_myobj2023-09-29T22:12:07.891Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:37.116 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.116 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.116 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.116 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.117 DEBUG:boto:Method: GET 2023-09-29T22:12:37.117 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.117 DEBUG:boto:Data: 2023-09-29T22:12:37.117 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.117 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.117 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.117 DEBUG:boto:Params: {} 2023-09-29T22:12:37.117 DEBUG:boto:Token: None 2023-09-29T22:12:37.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.118 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0= 2023-09-29T22:12:37.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0=', 'Content-Length': '0'} 2023-09-29T22:12:37.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091214acffcea7088-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.120 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.120 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.120 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.120 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.120 DEBUG:boto:Method: GET 2023-09-29T22:12:37.120 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.120 DEBUG:boto:Data: 2023-09-29T22:12:37.120 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.121 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.121 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.121 DEBUG:boto:Params: {} 2023-09-29T22:12:37.121 DEBUG:boto:Token: None 2023-09-29T22:12:37.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0= 2023-09-29T22:12:37.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0=', 'Content-Length': '0'} 2023-09-29T22:12:37.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a0a79b72350d460-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.166 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.167 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.167 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.167 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.167 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.167 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.167 DEBUG:boto:Data: 2023-09-29T22:12:37.167 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.167 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.167 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.168 DEBUG:boto:Params: {} 2023-09-29T22:12:37.168 DEBUG:boto:Token: None 2023-09-29T22:12:37.168 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/NFBg9WSTWrxLPwAe3NbGLKchcM= 2023-09-29T22:12:37.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/NFBg9WSTWrxLPwAe3NbGLKchcM=', 'Content-Length': '0'} 2023-09-29T22:12:37.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e2eac459ba70570-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.170 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.170 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.170 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.170 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.170 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.170 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.170 DEBUG:boto:Data: 2023-09-29T22:12:37.171 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.171 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.171 DEBUG:boto:Params: {} 2023-09-29T22:12:37.171 DEBUG:boto:Token: None 2023-09-29T22:12:37.171 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.171 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/NFBg9WSTWrxLPwAe3NbGLKchcM= 2023-09-29T22:12:37.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/NFBg9WSTWrxLPwAe3NbGLKchcM=', 'Content-Length': '0'} 2023-09-29T22:12:37.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c382af6c63aea1f-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:37.174 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.174 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.174 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.174 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.174 DEBUG:boto:Method: GET 2023-09-29T22:12:37.174 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.174 DEBUG:boto:Data: 2023-09-29T22:12:37.174 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.175 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.175 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.175 DEBUG:boto:Params: {} 2023-09-29T22:12:37.175 DEBUG:boto:Token: None 2023-09-29T22:12:37.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0= 2023-09-29T22:12:37.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0=', 'Content-Length': '0'} 2023-09-29T22:12:37.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039c84e4cb3f8c7d3-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.177 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.177 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.177 DEBUG:boto:path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.178 DEBUG:boto:auth_path=/dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.178 DEBUG:boto:Method: GET 2023-09-29T22:12:37.178 DEBUG:boto:Path: /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.178 DEBUG:boto:Data: 2023-09-29T22:12:37.178 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.178 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.178 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.178 DEBUG:boto:Params: {} 2023-09-29T22:12:37.178 DEBUG:boto:Token: None 2023-09-29T22:12:37.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj?versionId=null 2023-09-29T22:12:37.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0= 2023-09-29T22:12:37.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pB8Xvq8ECpwMUbYRpWl/33Ylbk0=', 'Content-Length': '0'} 2023-09-29T22:12:37.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc02172bea7a0b59-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.181 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.181 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.181 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.182 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.182 DEBUG:boto:Method: GET 2023-09-29T22:12:37.182 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.182 DEBUG:boto:Data: 2023-09-29T22:12:37.182 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.182 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.182 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.182 DEBUG:boto:Params: {} 2023-09-29T22:12:37.182 DEBUG:boto:Token: None 2023-09-29T22:12:37.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006640a02e20a7886c-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.185 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.185 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.185 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.185 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.186 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.186 DEBUG:boto:Method: GET 2023-09-29T22:12:37.186 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.186 DEBUG:boto:Data: 2023-09-29T22:12:37.186 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.186 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.186 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.186 DEBUG:boto:Params: {} 2023-09-29T22:12:37.186 DEBUG:boto:Token: None 2023-09-29T22:12:37.187 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.187 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050ef5fe2e3443a3b-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.189 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.190 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.190 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.190 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.190 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:37.190 DEBUG:boto:Data: 2023-09-29T22:12:37.190 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.190 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.190 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.191 DEBUG:boto:Params: {} 2023-09-29T22:12:37.191 DEBUG:boto:Token: None 2023-09-29T22:12:37.191 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj 2023-09-29T22:12:37.191 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F3TOSg5PEqlvGukcwRJK4EkS65U= 2023-09-29T22:12:37.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F3TOSg5PEqlvGukcwRJK4EkS65U=', 'Content-Length': '0'} 2023-09-29T22:12:37.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adfd868c07254820-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.193 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.193 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.193 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.193 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:37.193 DEBUG:boto:Data: 2023-09-29T22:12:37.193 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.194 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.194 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.194 DEBUG:boto:Params: {} 2023-09-29T22:12:37.194 DEBUG:boto:Token: None 2023-09-29T22:12:37.194 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj 2023-09-29T22:12:37.194 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F3TOSg5PEqlvGukcwRJK4EkS65U= 2023-09-29T22:12:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F3TOSg5PEqlvGukcwRJK4EkS65U=', 'Content-Length': '0'} 2023-09-29T22:12:37.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079af2a8c4f69f72c-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:12:37.196 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.196 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.197 DEBUG:boto:Method: GET 2023-09-29T22:12:37.197 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:37.197 DEBUG:boto:Data: 2023-09-29T22:12:37.197 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.197 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.197 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.197 DEBUG:boto:Params: {} 2023-09-29T22:12:37.197 DEBUG:boto:Token: None 2023-09-29T22:12:37.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj 2023-09-29T22:12:37.198 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oBGbT9M67zzY0+gsOGAqPEBRtpo= 2023-09-29T22:12:37.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oBGbT9M67zzY0+gsOGAqPEBRtpo=', 'Content-Length': '0'} 2023-09-29T22:12:37.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000450ff7cda26d1756-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.200 DEBUG:boto:path=/dmuhsu-12/myobj 2023-09-29T22:12:37.200 DEBUG:boto:auth_path=/dmuhsu-12/myobj 2023-09-29T22:12:37.200 DEBUG:boto:Method: GET 2023-09-29T22:12:37.200 DEBUG:boto:Path: /dmuhsu-12/myobj 2023-09-29T22:12:37.200 DEBUG:boto:Data: 2023-09-29T22:12:37.200 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.200 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.200 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.200 DEBUG:boto:Params: {} 2023-09-29T22:12:37.201 DEBUG:boto:Token: None 2023-09-29T22:12:37.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/myobj 2023-09-29T22:12:37.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oBGbT9M67zzY0+gsOGAqPEBRtpo= 2023-09-29T22:12:37.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oBGbT9M67zzY0+gsOGAqPEBRtpo=', 'Content-Length': '0'} 2023-09-29T22:12:37.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7437ad64160b2cc-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.203 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.203 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.204 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.204 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.204 DEBUG:boto:Method: GET 2023-09-29T22:12:37.204 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.204 DEBUG:boto:Data: 2023-09-29T22:12:37.204 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.204 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.204 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.204 DEBUG:boto:Params: {} 2023-09-29T22:12:37.205 DEBUG:boto:Token: None 2023-09-29T22:12:37.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.205 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f9be9c3ebb34dea-0065174bd5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.207 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.207 DEBUG:boto:path=/dmuhsu-12/ 2023-09-29T22:12:37.208 DEBUG:boto:auth_path=/dmuhsu-12/ 2023-09-29T22:12:37.208 DEBUG:boto:path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.208 DEBUG:boto:auth_path=/dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.208 DEBUG:boto:Method: GET 2023-09-29T22:12:37.208 DEBUG:boto:Path: /dmuhsu-12/?prefix=myobj 2023-09-29T22:12:37.208 DEBUG:boto:Data: 2023-09-29T22:12:37.208 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.208 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.208 DEBUG:boto:Params: {} 2023-09-29T22:12:37.209 DEBUG:boto:Token: None 2023-09-29T22:12:37.209 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-12/ 2023-09-29T22:12:37.209 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE= 2023-09-29T22:12:37.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q+yp3yy0aiBIjWHqQjtboADcdSE=', 'Content-Length': '0'} 2023-09-29T22:12:37.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046297fc0045f7c10-0065174bd5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.212 DEBUG:boto:b'dmuhsu-12myobj1000falsemyobj2023-09-29T22:12:07.870Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:12:37.212 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-12 zones={test-zone2, test-zone3} 2023-09-29T22:12:37.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-09-29T22:12:37.213 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-13 2023-09-29T22:12:37.213 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:12:37.213 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:12:37.214 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.214 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:12:37.214 DEBUG:boto:Data: 2023-09-29T22:12:37.214 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.214 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:37.214 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.214 DEBUG:boto:Params: {} 2023-09-29T22:12:37.214 DEBUG:boto:Token: None 2023-09-29T22:12:37.214 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-13/ 2023-09-29T22:12:37.215 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0zTEYAnEjAU5Kr0/qaegPsc/NeQ= 2023-09-29T22:12:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0zTEYAnEjAU5Kr0/qaegPsc/NeQ=', 'Content-Length': '0'} 2023-09-29T22:12:37.229 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065b2b6fad94cc477-0065174bd5-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.230 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-14 2023-09-29T22:12:37.230 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:12:37.230 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:12:37.230 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.230 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:12:37.230 DEBUG:boto:Data: 2023-09-29T22:12:37.230 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.230 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.231 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.231 DEBUG:boto:Params: {} 2023-09-29T22:12:37.231 DEBUG:boto:Token: None 2023-09-29T22:12:37.231 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-14/ 2023-09-29T22:12:37.231 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/fAME9N3DU1E+lNp/U/hcrqw9uI= 2023-09-29T22:12:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/fAME9N3DU1E+lNp/U/hcrqw9uI=', 'Content-Length': '0'} 2023-09-29T22:12:37.256 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000879a2e6c9855477f-0065174bd5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.257 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-15 2023-09-29T22:12:37.257 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:12:37.257 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:12:37.257 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.257 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:12:37.257 DEBUG:boto:Data: 2023-09-29T22:12:37.257 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.258 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.258 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.258 DEBUG:boto:Params: {} 2023-09-29T22:12:37.258 DEBUG:boto:Token: None 2023-09-29T22:12:37.258 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-15/ 2023-09-29T22:12:37.258 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vR72XOrHpqtYLxx9ceSnqVn5YSc= 2023-09-29T22:12:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vR72XOrHpqtYLxx9ceSnqVn5YSc=', 'Content-Length': '0'} 2023-09-29T22:12:37.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038719bef73a35eec-0065174bd5-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.290 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:12:37.290 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:12:37.290 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.290 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:12:37.290 DEBUG:boto:Data: 2023-09-29T22:12:37.291 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:37.291 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.291 DEBUG:boto:Params: {} 2023-09-29T22:12:37.291 DEBUG:boto:Token: None 2023-09-29T22:12:37.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-13/ 2023-09-29T22:12:37.291 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tKy2JjnxQYI8gy2EYS5IxsOpv6M= 2023-09-29T22:12:37.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tKy2JjnxQYI8gy2EYS5IxsOpv6M=', 'Content-Length': '0'} 2023-09-29T22:12:37.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020eed788bd55cc08-0065174bd5-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.294 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:12:37.294 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:12:37.294 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.294 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:12:37.295 DEBUG:boto:Data: 2023-09-29T22:12:37.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:37.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:12:37.295 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.295 DEBUG:boto:Params: {} 2023-09-29T22:12:37.295 DEBUG:boto:Token: None 2023-09-29T22:12:37.295 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-13/myobj 2023-09-29T22:12:37.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qWslQ5QCBf9pqW0wGkHZ0qaJ6XA= 2023-09-29T22:12:37.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qWslQ5QCBf9pqW0wGkHZ0qaJ6XA='} 2023-09-29T22:12:37.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbf3a9b9244c1c2a-0065174bd5-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.301 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:12:37.302 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:12:37.302 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.302 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:12:37.302 DEBUG:boto:Data: 2023-09-29T22:12:37.302 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.302 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.302 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.302 DEBUG:boto:Params: {} 2023-09-29T22:12:37.303 DEBUG:boto:Token: None 2023-09-29T22:12:37.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-14/ 2023-09-29T22:12:37.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ONwvvKpIB+v08PO1JHotoje3l88= 2023-09-29T22:12:37.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ONwvvKpIB+v08PO1JHotoje3l88=', 'Content-Length': '0'} 2023-09-29T22:12:37.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000812acdb9bda7b427-0065174bd5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.305 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:12:37.305 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:12:37.305 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.306 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:12:37.306 DEBUG:boto:Data: 2023-09-29T22:12:37.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:37.306 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:12:37.306 DEBUG:boto:Port: 8000 2023-09-29T22:12:37.306 DEBUG:boto:Params: {} 2023-09-29T22:12:37.306 DEBUG:boto:Token: None 2023-09-29T22:12:37.306 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-14/myobj 2023-09-29T22:12:37.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lip4eoG84Xap8dGwD4vMMcS1gQ4= 2023-09-29T22:12:37.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lip4eoG84Xap8dGwD4vMMcS1gQ4='} 2023-09-29T22:12:37.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bc67095b2fac281-0065174bd5-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.312 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:12:37.312 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:12:37.312 DEBUG:boto:Method: HEAD 2023-09-29T22:12:37.313 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:12:37.313 DEBUG:boto:Data: 2023-09-29T22:12:37.313 DEBUG:boto:Headers: {} 2023-09-29T22:12:37.313 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.313 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.313 DEBUG:boto:Params: {} 2023-09-29T22:12:37.313 DEBUG:boto:Token: None 2023-09-29T22:12:37.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-15/ 2023-09-29T22:12:37.313 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BHFxld7b7xHcvis+yCCf+t3VyoM= 2023-09-29T22:12:37.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BHFxld7b7xHcvis+yCCf+t3VyoM=', 'Content-Length': '0'} 2023-09-29T22:12:37.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bff8770222bb8053-0065174bd5-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.316 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:12:37.316 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:12:37.316 DEBUG:boto:Method: PUT 2023-09-29T22:12:37.316 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:12:37.316 DEBUG:boto:Data: 2023-09-29T22:12:37.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:12:37.317 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:12:37.317 DEBUG:boto:Port: 8001 2023-09-29T22:12:37.317 DEBUG:boto:Params: {} 2023-09-29T22:12:37.317 DEBUG:boto:Token: None 2023-09-29T22:12:37.317 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:12:37 GMT /dmuhsu-15/myobj 2023-09-29T22:12:37.317 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s8C5I4xCM6X+7MjDPQ5j5EJ8mAA= 2023-09-29T22:12:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:12:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s8C5I4xCM6X+7MjDPQ5j5EJ8mAA='} 2023-09-29T22:12:37.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c060e0f72f2bfd8-0065174bd5-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:37.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.363 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:37.412 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:37.413 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:37.441 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:12:37.452 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025557.276011_82.1", 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:12:37.276011Z" 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.454 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:12:37.455 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025557.276011_82.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:37.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:12:37.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.456 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:37.501 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:37.502 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:37.530 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:37.530+0000 7fa0c3929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:37.531 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:37.531+0000 7fa0c3929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:37.542 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:12:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:12:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:12:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:37.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025557.276011_82.1", 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:37.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:12:37.276011Z", 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:12:37.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:12:37.548 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:12:37.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:12:37.549 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:12:37Z" 2023-09-29T22:12:37.549 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:12:37.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025557.276011_82.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:37.276011Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:37Z" } 2023-09-29T22:12:37.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025557.276011_82.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:37.276011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:37.550 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025557.276011_82.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:37.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025557.276011_82.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:37.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:12:37.550 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:12:37.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:37.551 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:37.596 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:37.596 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:37.624 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:37.625+0000 7fd9f2b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:37.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:12:37.625+0000 7fd9f2b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:37.636 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:12:37.636 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:12:37.636 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:37.637 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.638 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:37.639 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025557.276011_82.1", 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:37.640 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:12:37.276011Z", 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:12:37.641 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.642 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:12:37.643 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:12:37.643 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:12:37.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:12:37.643 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:12:37Z" 2023-09-29T22:12:37.643 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:12:37.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025557.276011_82.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:37.276011Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:37Z" } 2023-09-29T22:12:37.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025557.276011_82.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:37.276011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:12:37.644 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025557.276011_82.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:37.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025557.276011_82.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:12:37.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:12:37.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:37.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:12:37.645 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:12:37.696 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:37.696 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:37.728 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:37.728+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:12:42.729 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:42.729+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:12:47.729 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:47.729+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:12:52.730 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:52.730+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:12:57.731 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:12:57.730+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:02.731 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:02.731+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:07.732 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:07.732+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:12.733 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:12.733+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:17.734 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:17.734+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:22.734 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:22.734+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:27.735 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:27.735+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:32.736 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:32.735+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:37.736 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:37.736+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:42.737 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:42.737+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:47.738 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:47.738+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:52.738 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:52.738+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:13:57.739 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:13:57.739+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:14:02.740 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:02.740+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:14:07.740 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:07.740+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:14:12.741 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:12.741+0000 7fad9a125880 1 waiting to reach incremental sync.. 2023-09-29T22:14:17.743 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:17.743+0000 7fad9a125880 1 bucket sync caught up with source: 2023-09-29T22:14:17.743 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:17.743 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:17.743 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:17.743+0000 7fad9a125880 0 bucket checkpoint complete 2023-09-29T22:14:17.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-13 zones={test-zone2, test-zone1} 2023-09-29T22:14:17.759 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.759 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.760 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.760 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:17.760 DEBUG:boto:Data: 2023-09-29T22:14:17.760 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.760 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.760 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.760 DEBUG:boto:Params: {} 2023-09-29T22:14:17.761 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:14:17.761 DEBUG:boto:Token: None 2023-09-29T22:14:17.761 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.761 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TroNkxmmgbDHnfvZo48EYU0TqxI= 2023-09-29T22:14:17.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TroNkxmmgbDHnfvZo48EYU0TqxI=', 'Content-Length': '0'} 2023-09-29T22:14:17.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bb08207f3905e6c-0065174c39-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.766 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.766 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.766 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.766 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:17.766 DEBUG:boto:Data: 2023-09-29T22:14:17.767 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.767 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.767 DEBUG:boto:Params: {} 2023-09-29T22:14:17.767 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:14:17.767 DEBUG:boto:Token: None 2023-09-29T22:14:17.767 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TroNkxmmgbDHnfvZo48EYU0TqxI= 2023-09-29T22:14:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TroNkxmmgbDHnfvZo48EYU0TqxI=', 'Content-Length': '0'} 2023-09-29T22:14:17.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6936faa18c4fd31-0065174c39-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:17.772 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.772 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.772 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.772 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.772 DEBUG:boto:Method: GET 2023-09-29T22:14:17.772 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.772 DEBUG:boto:Data: 2023-09-29T22:14:17.773 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.773 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.773 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.773 DEBUG:boto:Params: {} 2023-09-29T22:14:17.773 DEBUG:boto:Token: None 2023-09-29T22:14:17.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/?versions 2023-09-29T22:14:17.773 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E= 2023-09-29T22:14:17.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E=', 'Content-Length': '0'} 2023-09-29T22:14:17.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007085caf87d8b90fe-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.776 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.777 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:17.777 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:17.777 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.777 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.777 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.777 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.777 DEBUG:boto:Method: GET 2023-09-29T22:14:17.777 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.778 DEBUG:boto:Data: 2023-09-29T22:14:17.778 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.778 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.778 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.778 DEBUG:boto:Params: {} 2023-09-29T22:14:17.778 DEBUG:boto:Token: None 2023-09-29T22:14:17.778 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/?versions 2023-09-29T22:14:17.779 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E= 2023-09-29T22:14:17.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E=', 'Content-Length': '0'} 2023-09-29T22:14:17.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c73011af30e86b-0065174c39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.781 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.781 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:17.782 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.782 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.783 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.783 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.783 DEBUG:boto:Method: GET 2023-09-29T22:14:17.783 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.783 DEBUG:boto:Data: 2023-09-29T22:14:17.784 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.784 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.784 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.784 DEBUG:boto:Params: {} 2023-09-29T22:14:17.784 DEBUG:boto:Token: None 2023-09-29T22:14:17.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/?versions 2023-09-29T22:14:17.785 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E= 2023-09-29T22:14:17.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E=', 'Content-Length': '0'} 2023-09-29T22:14:17.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b0dc4d11de6e79-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.787 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.787 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.787 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.788 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.788 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.788 DEBUG:boto:Method: GET 2023-09-29T22:14:17.788 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:17.788 DEBUG:boto:Data: 2023-09-29T22:14:17.788 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.788 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.788 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.789 DEBUG:boto:Params: {} 2023-09-29T22:14:17.789 DEBUG:boto:Token: None 2023-09-29T22:14:17.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/?versions 2023-09-29T22:14:17.789 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E= 2023-09-29T22:14:17.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k+fyLmjnhKMrrWm3Z01ALf00//E=', 'Content-Length': '0'} 2023-09-29T22:14:17.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002deefc4b12300117-0065174c39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.792 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:17.792 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.793 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.793 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.793 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.793 DEBUG:boto:Method: GET 2023-09-29T22:14:17.793 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.793 DEBUG:boto:Data: 2023-09-29T22:14:17.793 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.793 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.793 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.794 DEBUG:boto:Params: {} 2023-09-29T22:14:17.794 DEBUG:boto:Token: None 2023-09-29T22:14:17.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.794 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk= 2023-09-29T22:14:17.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk=', 'Content-Length': '0'} 2023-09-29T22:14:17.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76b8aa20d09c228-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.796 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.796 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.797 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.797 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.797 DEBUG:boto:Method: GET 2023-09-29T22:14:17.797 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.797 DEBUG:boto:Data: 2023-09-29T22:14:17.797 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.797 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.797 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.798 DEBUG:boto:Params: {} 2023-09-29T22:14:17.798 DEBUG:boto:Token: None 2023-09-29T22:14:17.798 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.798 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk= 2023-09-29T22:14:17.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk=', 'Content-Length': '0'} 2023-09-29T22:14:17.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000548647c1258c6e9-0065174c39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.800 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.801 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.801 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.801 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.801 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.801 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.801 DEBUG:boto:Data: 2023-09-29T22:14:17.801 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.801 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.802 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.802 DEBUG:boto:Params: {} 2023-09-29T22:14:17.802 DEBUG:boto:Token: None 2023-09-29T22:14:17.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Rm66Z4wtMA2nwKY6C3ikVcYHxF4= 2023-09-29T22:14:17.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Rm66Z4wtMA2nwKY6C3ikVcYHxF4=', 'Content-Length': '0'} 2023-09-29T22:14:17.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049f9d6f0bb36857e-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.804 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.804 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.804 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.805 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.805 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.805 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.805 DEBUG:boto:Data: 2023-09-29T22:14:17.805 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.805 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.805 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.806 DEBUG:boto:Params: {} 2023-09-29T22:14:17.806 DEBUG:boto:Token: None 2023-09-29T22:14:17.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Rm66Z4wtMA2nwKY6C3ikVcYHxF4= 2023-09-29T22:14:17.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Rm66Z4wtMA2nwKY6C3ikVcYHxF4=', 'Content-Length': '0'} 2023-09-29T22:14:17.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088cdca5bda0ba8d0-0065174c39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:17.808 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.809 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.809 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.809 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.809 DEBUG:boto:Method: GET 2023-09-29T22:14:17.809 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.809 DEBUG:boto:Data: 2023-09-29T22:14:17.809 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.809 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.809 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.810 DEBUG:boto:Params: {} 2023-09-29T22:14:17.810 DEBUG:boto:Token: None 2023-09-29T22:14:17.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.810 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk= 2023-09-29T22:14:17.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk=', 'Content-Length': '0'} 2023-09-29T22:14:17.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000634b80f5d6d2958e-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.812 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.812 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.812 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.812 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.813 DEBUG:boto:Method: GET 2023-09-29T22:14:17.813 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.813 DEBUG:boto:Data: 2023-09-29T22:14:17.813 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.813 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.813 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.813 DEBUG:boto:Params: {} 2023-09-29T22:14:17.813 DEBUG:boto:Token: None 2023-09-29T22:14:17.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:17.814 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk= 2023-09-29T22:14:17.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KbFvJIjukqlK3i27hwoLJt2kuxk=', 'Content-Length': '0'} 2023-09-29T22:14:17.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076d4432277b7d19a-0065174c39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.816 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.816 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.816 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.816 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.816 DEBUG:boto:Method: GET 2023-09-29T22:14:17.816 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.816 DEBUG:boto:Data: 2023-09-29T22:14:17.816 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.817 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.817 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.817 DEBUG:boto:Params: {} 2023-09-29T22:14:17.817 DEBUG:boto:Token: None 2023-09-29T22:14:17.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ= 2023-09-29T22:14:17.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ=', 'Content-Length': '0'} 2023-09-29T22:14:17.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001112999173f62bb8-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.819 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.820 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.820 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.820 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.820 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.820 DEBUG:boto:Method: GET 2023-09-29T22:14:17.821 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.821 DEBUG:boto:Data: 2023-09-29T22:14:17.821 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.821 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.821 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.821 DEBUG:boto:Params: {} 2023-09-29T22:14:17.821 DEBUG:boto:Token: None 2023-09-29T22:14:17.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.822 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ= 2023-09-29T22:14:17.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ=', 'Content-Length': '0'} 2023-09-29T22:14:17.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c702168f98295099-0065174c39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.824 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.824 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.825 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.825 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.825 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:17.825 DEBUG:boto:Data: 2023-09-29T22:14:17.825 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.825 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.825 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.825 DEBUG:boto:Params: {} 2023-09-29T22:14:17.826 DEBUG:boto:Token: None 2023-09-29T22:14:17.826 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj 2023-09-29T22:14:17.826 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z4lGL1FUznafpho49XXfrZF5A2k= 2023-09-29T22:14:17.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z4lGL1FUznafpho49XXfrZF5A2k=', 'Content-Length': '0'} 2023-09-29T22:14:17.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086fa35020d75ae92-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.828 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.828 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.828 DEBUG:boto:Method: HEAD 2023-09-29T22:14:17.828 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:17.828 DEBUG:boto:Data: 2023-09-29T22:14:17.829 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.829 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.829 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.829 DEBUG:boto:Params: {} 2023-09-29T22:14:17.829 DEBUG:boto:Token: None 2023-09-29T22:14:17.829 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj 2023-09-29T22:14:17.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z4lGL1FUznafpho49XXfrZF5A2k= 2023-09-29T22:14:17.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z4lGL1FUznafpho49XXfrZF5A2k=', 'Content-Length': '0'} 2023-09-29T22:14:17.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000791ea1e82fb68b30-0065174c39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:17.831 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.832 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.832 DEBUG:boto:Method: GET 2023-09-29T22:14:17.832 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:17.832 DEBUG:boto:Data: 2023-09-29T22:14:17.832 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.832 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.832 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.832 DEBUG:boto:Params: {} 2023-09-29T22:14:17.833 DEBUG:boto:Token: None 2023-09-29T22:14:17.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj 2023-09-29T22:14:17.833 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lq6TbkzFrRcyOn1mNvQQb/QYlhI= 2023-09-29T22:14:17.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lq6TbkzFrRcyOn1mNvQQb/QYlhI=', 'Content-Length': '0'} 2023-09-29T22:14:17.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93bdfc80ed5c92f-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.878 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:17.879 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:17.879 DEBUG:boto:Method: GET 2023-09-29T22:14:17.879 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:17.879 DEBUG:boto:Data: 2023-09-29T22:14:17.879 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.879 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.879 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.879 DEBUG:boto:Params: {} 2023-09-29T22:14:17.880 DEBUG:boto:Token: None 2023-09-29T22:14:17.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/myobj 2023-09-29T22:14:17.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lq6TbkzFrRcyOn1mNvQQb/QYlhI= 2023-09-29T22:14:17.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lq6TbkzFrRcyOn1mNvQQb/QYlhI=', 'Content-Length': '0'} 2023-09-29T22:14:17.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c30e717fc64465eb-0065174c39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.882 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.883 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.883 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.883 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.883 DEBUG:boto:Method: GET 2023-09-29T22:14:17.883 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.883 DEBUG:boto:Data: 2023-09-29T22:14:17.883 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.883 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:17.883 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.884 DEBUG:boto:Params: {} 2023-09-29T22:14:17.884 DEBUG:boto:Token: None 2023-09-29T22:14:17.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ= 2023-09-29T22:14:17.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ=', 'Content-Length': '0'} 2023-09-29T22:14:17.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa5bcf2359b875e-0065174c39-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.886 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.887 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:17.887 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:17.887 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.887 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.887 DEBUG:boto:Method: GET 2023-09-29T22:14:17.887 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:17.887 DEBUG:boto:Data: 2023-09-29T22:14:17.887 DEBUG:boto:Headers: {} 2023-09-29T22:14:17.888 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:17.888 DEBUG:boto:Port: 8000 2023-09-29T22:14:17.888 DEBUG:boto:Params: {} 2023-09-29T22:14:17.888 DEBUG:boto:Token: None 2023-09-29T22:14:17.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:17 GMT /dmuhsu-13/ 2023-09-29T22:14:17.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ= 2023-09-29T22:14:17.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EYw4D6xHnyClrKBZqlniDhwwaoQ=', 'Content-Length': '0'} 2023-09-29T22:14:17.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000386b79064e000c8c-0065174c39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:17.891 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:17.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-13 zones={test-zone2, test-zone1} 2023-09-29T22:14:17.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:17.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:17.892 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:17.952 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:17.952 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:17.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:17.991+0000 7f67ded20880 1 bucket sync caught up with source: 2023-09-29T22:14:17.991 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:17.991 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:17.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:17.991+0000 7f67ded20880 0 bucket checkpoint complete 2023-09-29T22:14:18.003 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-13 zones={test-zone3, test-zone1} 2023-09-29T22:14:18.003 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.004 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.004 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.004 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:18.004 DEBUG:boto:Data: 2023-09-29T22:14:18.004 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.004 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.004 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.004 DEBUG:boto:Params: {} 2023-09-29T22:14:18.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:14:18.005 DEBUG:boto:Token: None 2023-09-29T22:14:18.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3f1KCI6zLzdSq9jUoxDFNuLom18= 2023-09-29T22:14:18.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3f1KCI6zLzdSq9jUoxDFNuLom18=', 'Content-Length': '0'} 2023-09-29T22:14:18.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098536cc8b8cab4e4-0065174c3a-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.009 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.009 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.009 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.009 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:18.010 DEBUG:boto:Data: 2023-09-29T22:14:18.010 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.010 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.010 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.010 DEBUG:boto:Params: {} 2023-09-29T22:14:18.010 DEBUG:boto:Token: None 2023-09-29T22:14:18.010 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.010 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3f1KCI6zLzdSq9jUoxDFNuLom18= 2023-09-29T22:14:18.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3f1KCI6zLzdSq9jUoxDFNuLom18=', 'Content-Length': '0'} 2023-09-29T22:14:18.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b125cd0b8ce24d5-0065174c3a-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:18.013 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.013 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.013 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.013 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.013 DEBUG:boto:Method: GET 2023-09-29T22:14:18.013 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.013 DEBUG:boto:Data: 2023-09-29T22:14:18.013 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.013 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.013 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.014 DEBUG:boto:Params: {} 2023-09-29T22:14:18.014 DEBUG:boto:Token: None 2023-09-29T22:14:18.014 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/?versions 2023-09-29T22:14:18.014 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM= 2023-09-29T22:14:18.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM=', 'Content-Length': '0'} 2023-09-29T22:14:18.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b5e7f8b04b3c03-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.016 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.017 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:18.017 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.017 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.017 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.017 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.017 DEBUG:boto:Method: GET 2023-09-29T22:14:18.017 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.018 DEBUG:boto:Data: 2023-09-29T22:14:18.018 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.018 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.018 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.018 DEBUG:boto:Params: {} 2023-09-29T22:14:18.018 DEBUG:boto:Token: None 2023-09-29T22:14:18.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/?versions 2023-09-29T22:14:18.018 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM= 2023-09-29T22:14:18.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM=', 'Content-Length': '0'} 2023-09-29T22:14:18.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000725c085e26e4011-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.021 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.021 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.021 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.021 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.021 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.021 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.022 DEBUG:boto:Method: GET 2023-09-29T22:14:18.022 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.022 DEBUG:boto:Data: 2023-09-29T22:14:18.022 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.022 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.022 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.022 DEBUG:boto:Params: {} 2023-09-29T22:14:18.022 DEBUG:boto:Token: None 2023-09-29T22:14:18.022 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/?versions 2023-09-29T22:14:18.023 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM= 2023-09-29T22:14:18.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM=', 'Content-Length': '0'} 2023-09-29T22:14:18.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3858547b7e8040-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.025 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.025 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.025 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.025 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.025 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.026 DEBUG:boto:Method: GET 2023-09-29T22:14:18.026 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:18.026 DEBUG:boto:Data: 2023-09-29T22:14:18.026 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.026 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.026 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.026 DEBUG:boto:Params: {} 2023-09-29T22:14:18.026 DEBUG:boto:Token: None 2023-09-29T22:14:18.026 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/?versions 2023-09-29T22:14:18.027 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM= 2023-09-29T22:14:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EjYx5mFbujAHV1MOYgxdtEuiDiM=', 'Content-Length': '0'} 2023-09-29T22:14:18.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f31affa7189355-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.029 DEBUG:boto:b'dmuhsu-13999falsemyobjnulltrue2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.029 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.029 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.029 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.029 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.030 DEBUG:boto:Method: GET 2023-09-29T22:14:18.030 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.030 DEBUG:boto:Data: 2023-09-29T22:14:18.030 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.030 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.030 DEBUG:boto:Params: {} 2023-09-29T22:14:18.030 DEBUG:boto:Token: None 2023-09-29T22:14:18.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.031 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ= 2023-09-29T22:14:18.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ=', 'Content-Length': '0'} 2023-09-29T22:14:18.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9d0cd6def7e6dc4-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.032 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.032 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.033 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.033 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.033 DEBUG:boto:Method: GET 2023-09-29T22:14:18.033 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.033 DEBUG:boto:Data: 2023-09-29T22:14:18.033 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.033 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.033 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.033 DEBUG:boto:Params: {} 2023-09-29T22:14:18.033 DEBUG:boto:Token: None 2023-09-29T22:14:18.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ= 2023-09-29T22:14:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ=', 'Content-Length': '0'} 2023-09-29T22:14:18.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037fbb64b7d0c110b-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.036 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.036 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.036 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.036 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.037 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.037 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.037 DEBUG:boto:Data: 2023-09-29T22:14:18.037 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.037 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.037 DEBUG:boto:Params: {} 2023-09-29T22:14:18.037 DEBUG:boto:Token: None 2023-09-29T22:14:18.037 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ggP2byDhlnbooYcGVBa9EJQqJ/Q= 2023-09-29T22:14:18.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ggP2byDhlnbooYcGVBa9EJQqJ/Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e18347f55de34f0-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.039 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.040 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.040 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.040 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.040 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.040 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.040 DEBUG:boto:Data: 2023-09-29T22:14:18.040 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.040 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.040 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.040 DEBUG:boto:Params: {} 2023-09-29T22:14:18.041 DEBUG:boto:Token: None 2023-09-29T22:14:18.041 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.041 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ggP2byDhlnbooYcGVBa9EJQqJ/Q= 2023-09-29T22:14:18.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ggP2byDhlnbooYcGVBa9EJQqJ/Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9097ad7569d9a60-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.043 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.043 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.043 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.043 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.043 DEBUG:boto:Method: GET 2023-09-29T22:14:18.043 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.043 DEBUG:boto:Data: 2023-09-29T22:14:18.043 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.043 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.043 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.044 DEBUG:boto:Params: {} 2023-09-29T22:14:18.044 DEBUG:boto:Token: None 2023-09-29T22:14:18.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ= 2023-09-29T22:14:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ=', 'Content-Length': '0'} 2023-09-29T22:14:18.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d312bd989a734093-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.046 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.046 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.046 DEBUG:boto:path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.046 DEBUG:boto:auth_path=/dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.046 DEBUG:boto:Method: GET 2023-09-29T22:14:18.046 DEBUG:boto:Path: /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.046 DEBUG:boto:Data: 2023-09-29T22:14:18.046 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.047 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.047 DEBUG:boto:Params: {} 2023-09-29T22:14:18.047 DEBUG:boto:Token: None 2023-09-29T22:14:18.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj?versionId=null 2023-09-29T22:14:18.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ= 2023-09-29T22:14:18.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ktOEUW3TUm2AubMPlR8XMRShIOQ=', 'Content-Length': '0'} 2023-09-29T22:14:18.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001981e99dda19bd7c-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.049 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.049 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.049 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.050 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.050 DEBUG:boto:Method: GET 2023-09-29T22:14:18.050 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.050 DEBUG:boto:Data: 2023-09-29T22:14:18.050 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.050 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.050 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.050 DEBUG:boto:Params: {} 2023-09-29T22:14:18.050 DEBUG:boto:Token: None 2023-09-29T22:14:18.050 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.051 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q= 2023-09-29T22:14:18.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84ebc191106fb0f-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.053 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.053 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.053 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.054 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.054 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.054 DEBUG:boto:Method: GET 2023-09-29T22:14:18.054 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.054 DEBUG:boto:Data: 2023-09-29T22:14:18.054 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.054 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.054 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.054 DEBUG:boto:Params: {} 2023-09-29T22:14:18.054 DEBUG:boto:Token: None 2023-09-29T22:14:18.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q= 2023-09-29T22:14:18.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a9687e9f74bc22-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.057 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.057 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.057 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.057 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.058 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:18.058 DEBUG:boto:Data: 2023-09-29T22:14:18.058 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.058 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.058 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.058 DEBUG:boto:Params: {} 2023-09-29T22:14:18.058 DEBUG:boto:Token: None 2023-09-29T22:14:18.058 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj 2023-09-29T22:14:18.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+CSTTMZBkVskn2sVYFOEtUmtrc= 2023-09-29T22:14:18.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+CSTTMZBkVskn2sVYFOEtUmtrc=', 'Content-Length': '0'} 2023-09-29T22:14:18.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b3872a1c113c99-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.060 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.060 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.060 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.060 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:18.060 DEBUG:boto:Data: 2023-09-29T22:14:18.061 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.061 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.061 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.061 DEBUG:boto:Params: {} 2023-09-29T22:14:18.061 DEBUG:boto:Token: None 2023-09-29T22:14:18.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj 2023-09-29T22:14:18.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+CSTTMZBkVskn2sVYFOEtUmtrc= 2023-09-29T22:14:18.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+CSTTMZBkVskn2sVYFOEtUmtrc=', 'Content-Length': '0'} 2023-09-29T22:14:18.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000258957f821bb969-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.063 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.063 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.063 DEBUG:boto:Method: GET 2023-09-29T22:14:18.063 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:18.064 DEBUG:boto:Data: 2023-09-29T22:14:18.064 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.064 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.064 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.064 DEBUG:boto:Params: {} 2023-09-29T22:14:18.064 DEBUG:boto:Token: None 2023-09-29T22:14:18.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj 2023-09-29T22:14:18.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ucGTySI46Fh6AygPRYpe4NCjvUU= 2023-09-29T22:14:18.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ucGTySI46Fh6AygPRYpe4NCjvUU=', 'Content-Length': '0'} 2023-09-29T22:14:18.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004015dd7cb6aa03d1-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.066 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:18.066 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:18.066 DEBUG:boto:Method: GET 2023-09-29T22:14:18.066 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:18.066 DEBUG:boto:Data: 2023-09-29T22:14:18.066 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.067 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.067 DEBUG:boto:Params: {} 2023-09-29T22:14:18.067 DEBUG:boto:Token: None 2023-09-29T22:14:18.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/myobj 2023-09-29T22:14:18.067 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ucGTySI46Fh6AygPRYpe4NCjvUU= 2023-09-29T22:14:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ucGTySI46Fh6AygPRYpe4NCjvUU=', 'Content-Length': '0'} 2023-09-29T22:14:18.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036f6b745231a049e-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.110 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.110 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.111 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.111 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.111 DEBUG:boto:Method: GET 2023-09-29T22:14:18.111 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.111 DEBUG:boto:Data: 2023-09-29T22:14:18.111 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.111 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.111 DEBUG:boto:Params: {} 2023-09-29T22:14:18.112 DEBUG:boto:Token: None 2023-09-29T22:14:18.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q= 2023-09-29T22:14:18.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41e3953e815022a-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.114 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.114 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:18.115 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:18.115 DEBUG:boto:path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.115 DEBUG:boto:auth_path=/dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.115 DEBUG:boto:Method: GET 2023-09-29T22:14:18.115 DEBUG:boto:Path: /dmuhsu-13/?prefix=myobj 2023-09-29T22:14:18.115 DEBUG:boto:Data: 2023-09-29T22:14:18.115 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.115 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.115 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.115 DEBUG:boto:Params: {} 2023-09-29T22:14:18.116 DEBUG:boto:Token: None 2023-09-29T22:14:18.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-13/ 2023-09-29T22:14:18.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q= 2023-09-29T22:14:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC/VzUJkvlutBEw+3YAN5Hbiv4Q=', 'Content-Length': '0'} 2023-09-29T22:14:18.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db31bfa9517894d2-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.118 DEBUG:boto:b'dmuhsu-13myobj1000falsemyobj2023-09-29T22:12:37.298Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-13 zones={test-zone3, test-zone1} 2023-09-29T22:14:18.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.119 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:18.171 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:18.171 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.205+0000 7f68e4926880 1 bucket sync caught up with source: 2023-09-29T22:14:18.205 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.205 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.205+0000 7f68e4926880 0 bucket checkpoint complete 2023-09-29T22:14:18.217 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-14 zones={test-zone1, test-zone2} 2023-09-29T22:14:18.217 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.217 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.217 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.217 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:18.217 DEBUG:boto:Data: 2023-09-29T22:14:18.217 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.218 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.218 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.218 DEBUG:boto:Params: {} 2023-09-29T22:14:18.218 DEBUG:boto:Token: None 2023-09-29T22:14:18.218 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.218 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA= 2023-09-29T22:14:18.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA=', 'Content-Length': '0'} 2023-09-29T22:14:18.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f5c8887ca9d60b8-0065174c3a-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.220 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.221 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.221 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.221 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:18.221 DEBUG:boto:Data: 2023-09-29T22:14:18.221 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.221 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.221 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.221 DEBUG:boto:Params: {} 2023-09-29T22:14:18.221 DEBUG:boto:Token: None 2023-09-29T22:14:18.222 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.222 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA= 2023-09-29T22:14:18.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA=', 'Content-Length': '0'} 2023-09-29T22:14:18.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000243e16663f6953f4-0065174c3a-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.224 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:18.224 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.224 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.225 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.225 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.225 DEBUG:boto:Method: GET 2023-09-29T22:14:18.225 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.225 DEBUG:boto:Data: 2023-09-29T22:14:18.225 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.225 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.225 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.225 DEBUG:boto:Params: {} 2023-09-29T22:14:18.225 DEBUG:boto:Token: None 2023-09-29T22:14:18.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f54d96759c71fcb-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.228 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.228 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.228 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:18.229 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.229 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.229 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.229 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.229 DEBUG:boto:Method: GET 2023-09-29T22:14:18.229 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.229 DEBUG:boto:Data: 2023-09-29T22:14:18.229 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.229 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.229 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.230 DEBUG:boto:Params: {} 2023-09-29T22:14:18.230 DEBUG:boto:Token: None 2023-09-29T22:14:18.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.230 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd0bc38c2ab6c03-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.233 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.233 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.233 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.233 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.233 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.233 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.234 DEBUG:boto:Method: GET 2023-09-29T22:14:18.234 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.234 DEBUG:boto:Data: 2023-09-29T22:14:18.234 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.234 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.234 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.234 DEBUG:boto:Params: {} 2023-09-29T22:14:18.234 DEBUG:boto:Token: None 2023-09-29T22:14:18.234 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.235 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff46c138ff84bde-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.237 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.237 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.237 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.237 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.237 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.238 DEBUG:boto:Method: GET 2023-09-29T22:14:18.238 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.238 DEBUG:boto:Data: 2023-09-29T22:14:18.238 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.238 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.238 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.238 DEBUG:boto:Params: {} 2023-09-29T22:14:18.238 DEBUG:boto:Token: None 2023-09-29T22:14:18.238 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.239 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a7b9333e445322-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.241 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.242 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.242 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.242 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.242 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.242 DEBUG:boto:Method: GET 2023-09-29T22:14:18.242 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.242 DEBUG:boto:Data: 2023-09-29T22:14:18.242 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.242 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.242 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.243 DEBUG:boto:Params: {} 2023-09-29T22:14:18.243 DEBUG:boto:Token: None 2023-09-29T22:14:18.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.243 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfd089bc68c6bc0d-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.245 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.245 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.246 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.246 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.246 DEBUG:boto:Method: GET 2023-09-29T22:14:18.246 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.246 DEBUG:boto:Data: 2023-09-29T22:14:18.246 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.246 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.246 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.246 DEBUG:boto:Params: {} 2023-09-29T22:14:18.246 DEBUG:boto:Token: None 2023-09-29T22:14:18.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.247 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000d7649776bc9b28-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.249 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.249 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.249 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.250 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.250 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.250 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.250 DEBUG:boto:Data: 2023-09-29T22:14:18.250 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.250 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.250 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.250 DEBUG:boto:Params: {} 2023-09-29T22:14:18.253 DEBUG:boto:Token: None 2023-09-29T22:14:18.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.254 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs= 2023-09-29T22:14:18.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs=', 'Content-Length': '0'} 2023-09-29T22:14:18.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8fe206f841139ac-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.255 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.255 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.256 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.256 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.256 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.256 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.256 DEBUG:boto:Data: 2023-09-29T22:14:18.256 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.256 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.256 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.256 DEBUG:boto:Params: {} 2023-09-29T22:14:18.257 DEBUG:boto:Token: None 2023-09-29T22:14:18.257 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.257 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs= 2023-09-29T22:14:18.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs=', 'Content-Length': '0'} 2023-09-29T22:14:18.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000439f315d8248697c-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.259 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.259 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.259 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.259 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.259 DEBUG:boto:Method: GET 2023-09-29T22:14:18.259 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.259 DEBUG:boto:Data: 2023-09-29T22:14:18.260 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.260 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.260 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.260 DEBUG:boto:Params: {} 2023-09-29T22:14:18.260 DEBUG:boto:Token: None 2023-09-29T22:14:18.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ee27f54b4a36a07-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.302 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.303 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.303 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.303 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.303 DEBUG:boto:Method: GET 2023-09-29T22:14:18.303 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.303 DEBUG:boto:Data: 2023-09-29T22:14:18.303 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.303 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.303 DEBUG:boto:Params: {} 2023-09-29T22:14:18.304 DEBUG:boto:Token: None 2023-09-29T22:14:18.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000546506d217b2cbdd-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.306 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.306 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.307 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.307 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.307 DEBUG:boto:Method: GET 2023-09-29T22:14:18.307 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.307 DEBUG:boto:Data: 2023-09-29T22:14:18.307 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.307 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.307 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.307 DEBUG:boto:Params: {} 2023-09-29T22:14:18.307 DEBUG:boto:Token: None 2023-09-29T22:14:18.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfcfe890761e90eb-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.310 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.310 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.311 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.311 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.311 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.311 DEBUG:boto:Method: GET 2023-09-29T22:14:18.311 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.311 DEBUG:boto:Data: 2023-09-29T22:14:18.311 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.311 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.311 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.311 DEBUG:boto:Params: {} 2023-09-29T22:14:18.312 DEBUG:boto:Token: None 2023-09-29T22:14:18.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000174630cb254296d9-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.314 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.315 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.315 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.315 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.315 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.315 DEBUG:boto:Data: 2023-09-29T22:14:18.315 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.316 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.316 DEBUG:boto:Params: {} 2023-09-29T22:14:18.316 DEBUG:boto:Token: None 2023-09-29T22:14:18.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM= 2023-09-29T22:14:18.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM=', 'Content-Length': '0'} 2023-09-29T22:14:18.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d1ab8857628035e-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.318 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.318 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.318 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.318 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.318 DEBUG:boto:Data: 2023-09-29T22:14:18.318 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.318 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.319 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.319 DEBUG:boto:Params: {} 2023-09-29T22:14:18.319 DEBUG:boto:Token: None 2023-09-29T22:14:18.319 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM= 2023-09-29T22:14:18.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM=', 'Content-Length': '0'} 2023-09-29T22:14:18.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007afdbbbee46cc89c-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.321 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.321 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.322 DEBUG:boto:Method: GET 2023-09-29T22:14:18.322 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.322 DEBUG:boto:Data: 2023-09-29T22:14:18.322 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.322 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.322 DEBUG:boto:Params: {} 2023-09-29T22:14:18.322 DEBUG:boto:Token: None 2023-09-29T22:14:18.322 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A= 2023-09-29T22:14:18.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A=', 'Content-Length': '0'} 2023-09-29T22:14:18.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecb392fdd0243c5a-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.370 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.371 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.371 DEBUG:boto:Method: GET 2023-09-29T22:14:18.371 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.371 DEBUG:boto:Data: 2023-09-29T22:14:18.371 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.371 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.371 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.371 DEBUG:boto:Params: {} 2023-09-29T22:14:18.371 DEBUG:boto:Token: None 2023-09-29T22:14:18.371 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A= 2023-09-29T22:14:18.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A=', 'Content-Length': '0'} 2023-09-29T22:14:18.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005412b0be5dd9ef66-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.374 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.374 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.374 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.374 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.374 DEBUG:boto:Method: GET 2023-09-29T22:14:18.375 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.375 DEBUG:boto:Data: 2023-09-29T22:14:18.375 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.375 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.375 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.375 DEBUG:boto:Params: {} 2023-09-29T22:14:18.375 DEBUG:boto:Token: None 2023-09-29T22:14:18.375 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.375 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59e3f6a9a352f7a-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.418 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.419 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.419 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.419 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.419 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.419 DEBUG:boto:Method: GET 2023-09-29T22:14:18.419 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.419 DEBUG:boto:Data: 2023-09-29T22:14:18.419 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.420 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.420 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.420 DEBUG:boto:Params: {} 2023-09-29T22:14:18.420 DEBUG:boto:Token: None 2023-09-29T22:14:18.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b95778b9d612df11-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.422 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-14 zones={test-zone1, test-zone2} 2023-09-29T22:14:18.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.423 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:18.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:18.476 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:18.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.511+0000 7f549cf36880 1 bucket sync caught up with source: 2023-09-29T22:14:18.511 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.511 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.511+0000 7f549cf36880 0 bucket checkpoint complete 2023-09-29T22:14:18.522 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-14 zones={test-zone3, test-zone2} 2023-09-29T22:14:18.522 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.523 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.523 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.523 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:18.523 DEBUG:boto:Data: 2023-09-29T22:14:18.523 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.523 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.523 DEBUG:boto:Params: {} 2023-09-29T22:14:18.523 DEBUG:boto:Token: None 2023-09-29T22:14:18.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA= 2023-09-29T22:14:18.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA=', 'Content-Length': '0'} 2023-09-29T22:14:18.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092f5f162559f9042-0065174c3a-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.526 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.526 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.526 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.526 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:18.527 DEBUG:boto:Data: 2023-09-29T22:14:18.527 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.527 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.527 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.527 DEBUG:boto:Params: {} 2023-09-29T22:14:18.527 DEBUG:boto:Token: None 2023-09-29T22:14:18.527 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.527 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA= 2023-09-29T22:14:18.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHfzfXws4dnFNJFfRzWCCiOhAfA=', 'Content-Length': '0'} 2023-09-29T22:14:18.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007211cab193df290e-0065174c3a-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:18.530 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.530 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.530 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.530 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.530 DEBUG:boto:Method: GET 2023-09-29T22:14:18.530 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.530 DEBUG:boto:Data: 2023-09-29T22:14:18.531 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.531 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.531 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.531 DEBUG:boto:Params: {} 2023-09-29T22:14:18.531 DEBUG:boto:Token: None 2023-09-29T22:14:18.531 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.531 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007043d3d829047fd5-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.533 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.534 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.534 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:18.534 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.534 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.534 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.535 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.535 DEBUG:boto:Method: GET 2023-09-29T22:14:18.535 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.535 DEBUG:boto:Data: 2023-09-29T22:14:18.535 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.535 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.535 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.535 DEBUG:boto:Params: {} 2023-09-29T22:14:18.535 DEBUG:boto:Token: None 2023-09-29T22:14:18.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7ec5ec07ed29e94-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.538 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.539 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.539 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.539 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.539 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.539 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.539 DEBUG:boto:Method: GET 2023-09-29T22:14:18.539 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.539 DEBUG:boto:Data: 2023-09-29T22:14:18.539 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.540 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.540 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.540 DEBUG:boto:Params: {} 2023-09-29T22:14:18.540 DEBUG:boto:Token: None 2023-09-29T22:14:18.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052fa9f06bbd3e8bc-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.542 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.543 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.543 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.543 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.543 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.543 DEBUG:boto:Method: GET 2023-09-29T22:14:18.543 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:18.543 DEBUG:boto:Data: 2023-09-29T22:14:18.543 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.543 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.543 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.544 DEBUG:boto:Params: {} 2023-09-29T22:14:18.544 DEBUG:boto:Token: None 2023-09-29T22:14:18.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/?versions 2023-09-29T22:14:18.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM= 2023-09-29T22:14:18.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfizK0iRZ6OY6JD6rGNxmi6yPOM=', 'Content-Length': '0'} 2023-09-29T22:14:18.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f558b81d5524a0-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.594 DEBUG:boto:b'dmuhsu-14999falsemyobjnulltrue2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.595 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.595 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.595 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.595 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.596 DEBUG:boto:Method: GET 2023-09-29T22:14:18.596 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.596 DEBUG:boto:Data: 2023-09-29T22:14:18.596 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.596 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.596 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.596 DEBUG:boto:Params: {} 2023-09-29T22:14:18.596 DEBUG:boto:Token: None 2023-09-29T22:14:18.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.597 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000870efb04b4ca32e0-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.598 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.599 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.599 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.599 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.599 DEBUG:boto:Method: GET 2023-09-29T22:14:18.599 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.599 DEBUG:boto:Data: 2023-09-29T22:14:18.599 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.599 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.599 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.600 DEBUG:boto:Params: {} 2023-09-29T22:14:18.600 DEBUG:boto:Token: None 2023-09-29T22:14:18.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.600 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021af075c4d5cda70-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.602 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.602 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.602 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.602 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.602 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.603 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.603 DEBUG:boto:Data: 2023-09-29T22:14:18.603 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.603 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.603 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.603 DEBUG:boto:Params: {} 2023-09-29T22:14:18.603 DEBUG:boto:Token: None 2023-09-29T22:14:18.603 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.603 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs= 2023-09-29T22:14:18.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs=', 'Content-Length': '0'} 2023-09-29T22:14:18.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb25bbc768022b52-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.605 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.605 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.605 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.605 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.605 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.606 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.606 DEBUG:boto:Data: 2023-09-29T22:14:18.606 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.606 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.606 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.606 DEBUG:boto:Params: {} 2023-09-29T22:14:18.606 DEBUG:boto:Token: None 2023-09-29T22:14:18.606 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs= 2023-09-29T22:14:18.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vm609WlIeVBujtoWSKDZqOpClxs=', 'Content-Length': '0'} 2023-09-29T22:14:18.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d80ced511137a8a-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.609 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.609 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.609 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.609 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.609 DEBUG:boto:Method: GET 2023-09-29T22:14:18.609 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.609 DEBUG:boto:Data: 2023-09-29T22:14:18.609 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.609 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.610 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.610 DEBUG:boto:Params: {} 2023-09-29T22:14:18.610 DEBUG:boto:Token: None 2023-09-29T22:14:18.610 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000971c3bf6a0dadf4b-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.612 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.612 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.612 DEBUG:boto:path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.612 DEBUG:boto:auth_path=/dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.612 DEBUG:boto:Method: GET 2023-09-29T22:14:18.612 DEBUG:boto:Path: /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.613 DEBUG:boto:Data: 2023-09-29T22:14:18.613 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.613 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.613 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.613 DEBUG:boto:Params: {} 2023-09-29T22:14:18.613 DEBUG:boto:Token: None 2023-09-29T22:14:18.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj?versionId=null 2023-09-29T22:14:18.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw= 2023-09-29T22:14:18.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U8cmNMmIf1mpSZHoX6fMMmhtwfw=', 'Content-Length': '0'} 2023-09-29T22:14:18.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea5df6075412758e-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.615 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.616 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.616 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.616 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.616 DEBUG:boto:Method: GET 2023-09-29T22:14:18.616 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.616 DEBUG:boto:Data: 2023-09-29T22:14:18.616 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.616 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.616 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.616 DEBUG:boto:Params: {} 2023-09-29T22:14:18.617 DEBUG:boto:Token: None 2023-09-29T22:14:18.617 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.617 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000549cce3432d9d610-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.619 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.619 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.619 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.620 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.620 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.620 DEBUG:boto:Method: GET 2023-09-29T22:14:18.620 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.620 DEBUG:boto:Data: 2023-09-29T22:14:18.620 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.620 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.620 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.620 DEBUG:boto:Params: {} 2023-09-29T22:14:18.620 DEBUG:boto:Token: None 2023-09-29T22:14:18.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.621 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a04f9cd91449135a-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.623 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.623 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.624 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.624 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.624 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.624 DEBUG:boto:Data: 2023-09-29T22:14:18.624 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.624 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.624 DEBUG:boto:Params: {} 2023-09-29T22:14:18.624 DEBUG:boto:Token: None 2023-09-29T22:14:18.625 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM= 2023-09-29T22:14:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM=', 'Content-Length': '0'} 2023-09-29T22:14:18.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad86ececb440410b-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.627 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.627 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.627 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.627 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.627 DEBUG:boto:Data: 2023-09-29T22:14:18.627 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.627 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.627 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.627 DEBUG:boto:Params: {} 2023-09-29T22:14:18.628 DEBUG:boto:Token: None 2023-09-29T22:14:18.628 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.628 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM= 2023-09-29T22:14:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3V8HASTaRWvEsfW5xOD8odAkUsM=', 'Content-Length': '0'} 2023-09-29T22:14:18.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a6053edaaccd291-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.630 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.630 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.630 DEBUG:boto:Method: GET 2023-09-29T22:14:18.630 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.630 DEBUG:boto:Data: 2023-09-29T22:14:18.630 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.630 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.631 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.631 DEBUG:boto:Params: {} 2023-09-29T22:14:18.631 DEBUG:boto:Token: None 2023-09-29T22:14:18.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.631 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A= 2023-09-29T22:14:18.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A=', 'Content-Length': '0'} 2023-09-29T22:14:18.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fea93dae377ff74b-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.674 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:18.674 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:18.675 DEBUG:boto:Method: GET 2023-09-29T22:14:18.675 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:18.675 DEBUG:boto:Data: 2023-09-29T22:14:18.675 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.675 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.675 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.675 DEBUG:boto:Params: {} 2023-09-29T22:14:18.675 DEBUG:boto:Token: None 2023-09-29T22:14:18.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/myobj 2023-09-29T22:14:18.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A= 2023-09-29T22:14:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:erZNFGcrw5Dij474/AasW5b2+2A=', 'Content-Length': '0'} 2023-09-29T22:14:18.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e350d5a912b6f927-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.678 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.678 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.678 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.678 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.679 DEBUG:boto:Method: GET 2023-09-29T22:14:18.679 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.679 DEBUG:boto:Data: 2023-09-29T22:14:18.679 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.679 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.679 DEBUG:boto:Params: {} 2023-09-29T22:14:18.679 DEBUG:boto:Token: None 2023-09-29T22:14:18.679 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.679 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc4dee3a4f719ac0-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.682 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.682 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:18.682 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:18.683 DEBUG:boto:path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.683 DEBUG:boto:auth_path=/dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.683 DEBUG:boto:Method: GET 2023-09-29T22:14:18.683 DEBUG:boto:Path: /dmuhsu-14/?prefix=myobj 2023-09-29T22:14:18.683 DEBUG:boto:Data: 2023-09-29T22:14:18.683 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.683 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:18.683 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.683 DEBUG:boto:Params: {} 2023-09-29T22:14:18.683 DEBUG:boto:Token: None 2023-09-29T22:14:18.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-14/ 2023-09-29T22:14:18.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8= 2023-09-29T22:14:18.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qFozO1qty8waYNXvnkB4+CZrKT8=', 'Content-Length': '0'} 2023-09-29T22:14:18.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88b225aeacf3960-0065174c3a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.686 DEBUG:boto:b'dmuhsu-14myobj1000falsemyobj2023-09-29T22:12:37.309Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.687 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-14 zones={test-zone3, test-zone2} 2023-09-29T22:14:18.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.687 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:18.740 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:18.741 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:18.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.774+0000 7fabe8529880 1 bucket sync caught up with source: 2023-09-29T22:14:18.774 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.774 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:18.774+0000 7fabe8529880 0 bucket checkpoint complete 2023-09-29T22:14:18.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-15 zones={test-zone1, test-zone3} 2023-09-29T22:14:18.786 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.786 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.786 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.786 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:14:18.786 DEBUG:boto:Data: 2023-09-29T22:14:18.786 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.786 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.786 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.787 DEBUG:boto:Params: {} 2023-09-29T22:14:18.787 DEBUG:boto:Token: None 2023-09-29T22:14:18.787 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.787 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2Eq5Fh2LtCVF2HB07v4QZW8cs8s= 2023-09-29T22:14:18.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2Eq5Fh2LtCVF2HB07v4QZW8cs8s=', 'Content-Length': '0'} 2023-09-29T22:14:18.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6815749911133d9-0065174c3a-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.790 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.790 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.790 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.790 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:14:18.790 DEBUG:boto:Data: 2023-09-29T22:14:18.790 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.790 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.790 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.791 DEBUG:boto:Params: {} 2023-09-29T22:14:18.791 DEBUG:boto:Token: None 2023-09-29T22:14:18.791 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.791 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2Eq5Fh2LtCVF2HB07v4QZW8cs8s= 2023-09-29T22:14:18.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2Eq5Fh2LtCVF2HB07v4QZW8cs8s=', 'Content-Length': '0'} 2023-09-29T22:14:18.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046773832e03964f8-0065174c3a-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:18.793 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.794 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.794 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.794 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.794 DEBUG:boto:Method: GET 2023-09-29T22:14:18.794 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.794 DEBUG:boto:Data: 2023-09-29T22:14:18.794 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.794 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.794 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.794 DEBUG:boto:Params: {} 2023-09-29T22:14:18.795 DEBUG:boto:Token: None 2023-09-29T22:14:18.795 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/?versions 2023-09-29T22:14:18.795 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U= 2023-09-29T22:14:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U=', 'Content-Length': '0'} 2023-09-29T22:14:18.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106aeabcf506e41f-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.797 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.797 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.797 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:18.798 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.798 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.798 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.798 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.798 DEBUG:boto:Method: GET 2023-09-29T22:14:18.798 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.798 DEBUG:boto:Data: 2023-09-29T22:14:18.798 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.798 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.799 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.799 DEBUG:boto:Params: {} 2023-09-29T22:14:18.799 DEBUG:boto:Token: None 2023-09-29T22:14:18.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/?versions 2023-09-29T22:14:18.799 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U= 2023-09-29T22:14:18.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U=', 'Content-Length': '0'} 2023-09-29T22:14:18.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c939d3ccc44a82-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.801 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.801 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:18.802 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.802 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.802 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.802 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.802 DEBUG:boto:Method: GET 2023-09-29T22:14:18.802 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.802 DEBUG:boto:Data: 2023-09-29T22:14:18.802 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.802 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.803 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.803 DEBUG:boto:Params: {} 2023-09-29T22:14:18.803 DEBUG:boto:Token: None 2023-09-29T22:14:18.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/?versions 2023-09-29T22:14:18.803 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U= 2023-09-29T22:14:18.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U=', 'Content-Length': '0'} 2023-09-29T22:14:18.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cbf8317cf7292ab-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.805 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.805 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.806 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.806 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.806 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.806 DEBUG:boto:Method: GET 2023-09-29T22:14:18.806 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:18.806 DEBUG:boto:Data: 2023-09-29T22:14:18.806 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.806 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.806 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.807 DEBUG:boto:Params: {} 2023-09-29T22:14:18.807 DEBUG:boto:Token: None 2023-09-29T22:14:18.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/?versions 2023-09-29T22:14:18.807 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U= 2023-09-29T22:14:18.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NHHw4zI+UjvANkbvCGiW2ar1j/U=', 'Content-Length': '0'} 2023-09-29T22:14:18.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003170fef49d21862f-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.809 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.810 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.810 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.810 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.810 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.810 DEBUG:boto:Method: GET 2023-09-29T22:14:18.810 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.810 DEBUG:boto:Data: 2023-09-29T22:14:18.810 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.810 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.810 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.811 DEBUG:boto:Params: {} 2023-09-29T22:14:18.811 DEBUG:boto:Token: None 2023-09-29T22:14:18.811 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.811 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU= 2023-09-29T22:14:18.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU=', 'Content-Length': '0'} 2023-09-29T22:14:18.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9829a3ab4dc512-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.813 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.813 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.813 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.813 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.813 DEBUG:boto:Method: GET 2023-09-29T22:14:18.814 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.814 DEBUG:boto:Data: 2023-09-29T22:14:18.814 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.814 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.814 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.814 DEBUG:boto:Params: {} 2023-09-29T22:14:18.814 DEBUG:boto:Token: None 2023-09-29T22:14:18.814 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.815 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU= 2023-09-29T22:14:18.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU=', 'Content-Length': '0'} 2023-09-29T22:14:18.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065b152fd22bb24af-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.816 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.816 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.817 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.817 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.817 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.817 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.817 DEBUG:boto:Data: 2023-09-29T22:14:18.817 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.817 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.817 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.817 DEBUG:boto:Params: {} 2023-09-29T22:14:18.817 DEBUG:boto:Token: None 2023-09-29T22:14:18.818 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.818 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xbiubZIGhll0AMpV3C4KK7E6kdM= 2023-09-29T22:14:18.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xbiubZIGhll0AMpV3C4KK7E6kdM=', 'Content-Length': '0'} 2023-09-29T22:14:18.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000325e9de8fa289750-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.819 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.820 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.820 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.820 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.820 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.820 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.820 DEBUG:boto:Data: 2023-09-29T22:14:18.820 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.820 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.821 DEBUG:boto:Params: {} 2023-09-29T22:14:18.821 DEBUG:boto:Token: None 2023-09-29T22:14:18.821 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.821 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xbiubZIGhll0AMpV3C4KK7E6kdM= 2023-09-29T22:14:18.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xbiubZIGhll0AMpV3C4KK7E6kdM=', 'Content-Length': '0'} 2023-09-29T22:14:18.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba283350e1e4148d-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.823 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.823 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.823 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.823 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.823 DEBUG:boto:Method: GET 2023-09-29T22:14:18.824 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.824 DEBUG:boto:Data: 2023-09-29T22:14:18.824 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.824 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.824 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.824 DEBUG:boto:Params: {} 2023-09-29T22:14:18.824 DEBUG:boto:Token: None 2023-09-29T22:14:18.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU= 2023-09-29T22:14:18.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU=', 'Content-Length': '0'} 2023-09-29T22:14:18.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdf8a3e8a5c26c67-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.826 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.826 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.826 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.827 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.827 DEBUG:boto:Method: GET 2023-09-29T22:14:18.827 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.827 DEBUG:boto:Data: 2023-09-29T22:14:18.827 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.827 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.827 DEBUG:boto:Params: {} 2023-09-29T22:14:18.827 DEBUG:boto:Token: None 2023-09-29T22:14:18.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:18.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU= 2023-09-29T22:14:18.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vgwj+dzcegSmvgIDyaDD3qjwwwU=', 'Content-Length': '0'} 2023-09-29T22:14:18.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005729ffc8b0092b1b-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.829 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.829 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.830 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.830 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.830 DEBUG:boto:Method: GET 2023-09-29T22:14:18.830 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.830 DEBUG:boto:Data: 2023-09-29T22:14:18.830 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.830 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.830 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.830 DEBUG:boto:Params: {} 2023-09-29T22:14:18.831 DEBUG:boto:Token: None 2023-09-29T22:14:18.831 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU= 2023-09-29T22:14:18.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU=', 'Content-Length': '0'} 2023-09-29T22:14:18.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3f3c887c431f9e-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.833 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.833 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.833 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.834 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.834 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.834 DEBUG:boto:Method: GET 2023-09-29T22:14:18.834 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.834 DEBUG:boto:Data: 2023-09-29T22:14:18.834 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.834 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.834 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.834 DEBUG:boto:Params: {} 2023-09-29T22:14:18.835 DEBUG:boto:Token: None 2023-09-29T22:14:18.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU= 2023-09-29T22:14:18.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU=', 'Content-Length': '0'} 2023-09-29T22:14:18.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac1b2d5a2812e7bd-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.837 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.837 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.837 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.838 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.838 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:18.838 DEBUG:boto:Data: 2023-09-29T22:14:18.838 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.838 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.838 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.838 DEBUG:boto:Params: {} 2023-09-29T22:14:18.838 DEBUG:boto:Token: None 2023-09-29T22:14:18.838 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj 2023-09-29T22:14:18.839 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xpWruy+PUi3crFRHu0PTYGYaHTI= 2023-09-29T22:14:18.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xpWruy+PUi3crFRHu0PTYGYaHTI=', 'Content-Length': '0'} 2023-09-29T22:14:18.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cbce12e55d53840-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.840 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.841 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.841 DEBUG:boto:Method: HEAD 2023-09-29T22:14:18.841 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:18.841 DEBUG:boto:Data: 2023-09-29T22:14:18.841 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.841 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.841 DEBUG:boto:Params: {} 2023-09-29T22:14:18.841 DEBUG:boto:Token: None 2023-09-29T22:14:18.842 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj 2023-09-29T22:14:18.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xpWruy+PUi3crFRHu0PTYGYaHTI= 2023-09-29T22:14:18.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xpWruy+PUi3crFRHu0PTYGYaHTI=', 'Content-Length': '0'} 2023-09-29T22:14:18.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000418632d89cd0c5e3-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:18.844 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.844 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.844 DEBUG:boto:Method: GET 2023-09-29T22:14:18.844 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:18.845 DEBUG:boto:Data: 2023-09-29T22:14:18.845 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.845 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.845 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.845 DEBUG:boto:Params: {} 2023-09-29T22:14:18.845 DEBUG:boto:Token: None 2023-09-29T22:14:18.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj 2023-09-29T22:14:18.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:89bV7hrEqvX3XFn1P5UuZTQHoRY= 2023-09-29T22:14:18.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:89bV7hrEqvX3XFn1P5UuZTQHoRY=', 'Content-Length': '0'} 2023-09-29T22:14:18.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ca129bfdef5092b-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.847 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:18.847 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:18.848 DEBUG:boto:Method: GET 2023-09-29T22:14:18.848 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:18.848 DEBUG:boto:Data: 2023-09-29T22:14:18.848 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.848 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.848 DEBUG:boto:Params: {} 2023-09-29T22:14:18.848 DEBUG:boto:Token: None 2023-09-29T22:14:18.848 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/myobj 2023-09-29T22:14:18.849 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:89bV7hrEqvX3XFn1P5UuZTQHoRY= 2023-09-29T22:14:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:89bV7hrEqvX3XFn1P5UuZTQHoRY=', 'Content-Length': '0'} 2023-09-29T22:14:18.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f545f64eb3476d8f-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.850 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.851 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.851 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.851 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.851 DEBUG:boto:Method: GET 2023-09-29T22:14:18.851 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.851 DEBUG:boto:Data: 2023-09-29T22:14:18.851 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.851 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:18.851 DEBUG:boto:Port: 8000 2023-09-29T22:14:18.851 DEBUG:boto:Params: {} 2023-09-29T22:14:18.852 DEBUG:boto:Token: None 2023-09-29T22:14:18.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU= 2023-09-29T22:14:18.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU=', 'Content-Length': '0'} 2023-09-29T22:14:18.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076f536225d970fbc-0065174c3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.854 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.855 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:18.855 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:18.855 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.855 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.855 DEBUG:boto:Method: GET 2023-09-29T22:14:18.855 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:18.855 DEBUG:boto:Data: 2023-09-29T22:14:18.855 DEBUG:boto:Headers: {} 2023-09-29T22:14:18.856 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:18.856 DEBUG:boto:Port: 8001 2023-09-29T22:14:18.856 DEBUG:boto:Params: {} 2023-09-29T22:14:18.856 DEBUG:boto:Token: None 2023-09-29T22:14:18.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:18 GMT /dmuhsu-15/ 2023-09-29T22:14:18.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU= 2023-09-29T22:14:18.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xFK+nhpORETTa8mKgbgl+MbUCVU=', 'Content-Length': '0'} 2023-09-29T22:14:18.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000337b4b9f795732b5-0065174c3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:18.898 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:18.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-15 zones={test-zone1, test-zone3} 2023-09-29T22:14:18.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:18.899 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:18.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:18.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:18.997 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:18.997+0000 7f3518536880 1 bucket sync caught up with source: 2023-09-29T22:14:18.997 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.997 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:18.997 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:18.997+0000 7f3518536880 0 bucket checkpoint complete 2023-09-29T22:14:19.010 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-15 zones={test-zone2, test-zone3} 2023-09-29T22:14:19.010 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.011 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.011 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.011 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:14:19.011 DEBUG:boto:Data: 2023-09-29T22:14:19.011 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.011 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.011 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.011 DEBUG:boto:Params: {} 2023-09-29T22:14:19.011 DEBUG:boto:Token: None 2023-09-29T22:14:19.012 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.012 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WEqK8k99mjg/O4xNdq0yVcplYi4= 2023-09-29T22:14:19.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WEqK8k99mjg/O4xNdq0yVcplYi4=', 'Content-Length': '0'} 2023-09-29T22:14:19.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c92c1fdf3b13b24c-0065174c3b-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.014 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.014 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.014 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.014 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:14:19.014 DEBUG:boto:Data: 2023-09-29T22:14:19.015 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.015 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.015 DEBUG:boto:Params: {} 2023-09-29T22:14:19.015 DEBUG:boto:Token: None 2023-09-29T22:14:19.015 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WEqK8k99mjg/O4xNdq0yVcplYi4= 2023-09-29T22:14:19.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WEqK8k99mjg/O4xNdq0yVcplYi4=', 'Content-Length': '0'} 2023-09-29T22:14:19.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c9fc87a09c7c6de-0065174c3b-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:19.018 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.018 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.018 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.018 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.018 DEBUG:boto:Method: GET 2023-09-29T22:14:19.018 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.019 DEBUG:boto:Data: 2023-09-29T22:14:19.019 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.019 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.019 DEBUG:boto:Params: {} 2023-09-29T22:14:19.019 DEBUG:boto:Token: None 2023-09-29T22:14:19.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/?versions 2023-09-29T22:14:19.019 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk= 2023-09-29T22:14:19.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk=', 'Content-Length': '0'} 2023-09-29T22:14:19.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bd4b730853b75c6-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.021 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.022 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:19.022 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:19.022 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.022 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.022 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.022 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.023 DEBUG:boto:Method: GET 2023-09-29T22:14:19.023 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.023 DEBUG:boto:Data: 2023-09-29T22:14:19.023 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.023 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.023 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.023 DEBUG:boto:Params: {} 2023-09-29T22:14:19.023 DEBUG:boto:Token: None 2023-09-29T22:14:19.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/?versions 2023-09-29T22:14:19.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk= 2023-09-29T22:14:19.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk=', 'Content-Length': '0'} 2023-09-29T22:14:19.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7414173dd699ad-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.026 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.026 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T22:14:19.026 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.026 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.026 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.026 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.026 DEBUG:boto:Method: GET 2023-09-29T22:14:19.027 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.027 DEBUG:boto:Data: 2023-09-29T22:14:19.027 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.027 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.027 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.027 DEBUG:boto:Params: {} 2023-09-29T22:14:19.027 DEBUG:boto:Token: None 2023-09-29T22:14:19.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/?versions 2023-09-29T22:14:19.027 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk= 2023-09-29T22:14:19.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk=', 'Content-Length': '0'} 2023-09-29T22:14:19.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767ec46c9f2a0ea6-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.030 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.030 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.030 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.030 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.030 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.030 DEBUG:boto:Method: GET 2023-09-29T22:14:19.031 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:14:19.031 DEBUG:boto:Data: 2023-09-29T22:14:19.031 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.031 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.031 DEBUG:boto:Params: {} 2023-09-29T22:14:19.031 DEBUG:boto:Token: None 2023-09-29T22:14:19.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/?versions 2023-09-29T22:14:19.031 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk= 2023-09-29T22:14:19.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H0spS6E9yVWRXErPn0kUzMdPexk=', 'Content-Length': '0'} 2023-09-29T22:14:19.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676d81ef6f26d010-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.034 DEBUG:boto:b'dmuhsu-15999falsemyobjnulltrue2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:19.034 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.037 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.037 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.037 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.037 DEBUG:boto:Method: GET 2023-09-29T22:14:19.037 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.037 DEBUG:boto:Data: 2023-09-29T22:14:19.038 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.038 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.038 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.038 DEBUG:boto:Params: {} 2023-09-29T22:14:19.038 DEBUG:boto:Token: None 2023-09-29T22:14:19.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ= 2023-09-29T22:14:19.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ=', 'Content-Length': '0'} 2023-09-29T22:14:19.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005739ef345c2862be-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.040 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.040 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.040 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.040 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.041 DEBUG:boto:Method: GET 2023-09-29T22:14:19.041 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.041 DEBUG:boto:Data: 2023-09-29T22:14:19.041 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.041 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.041 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.041 DEBUG:boto:Params: {} 2023-09-29T22:14:19.041 DEBUG:boto:Token: None 2023-09-29T22:14:19.041 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.041 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ= 2023-09-29T22:14:19.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ=', 'Content-Length': '0'} 2023-09-29T22:14:19.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001890f0c94520256-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.043 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.043 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.043 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.044 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.044 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.044 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.044 DEBUG:boto:Data: 2023-09-29T22:14:19.044 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.044 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.044 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.044 DEBUG:boto:Params: {} 2023-09-29T22:14:19.044 DEBUG:boto:Token: None 2023-09-29T22:14:19.045 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.045 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oNqPohcNXbW/johLhNBCajpwl7Q= 2023-09-29T22:14:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oNqPohcNXbW/johLhNBCajpwl7Q=', 'Content-Length': '0'} 2023-09-29T22:14:19.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023e88be6e0d00c41-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.046 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.047 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.047 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.047 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.047 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.047 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.047 DEBUG:boto:Data: 2023-09-29T22:14:19.047 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.047 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.047 DEBUG:boto:Params: {} 2023-09-29T22:14:19.047 DEBUG:boto:Token: None 2023-09-29T22:14:19.048 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oNqPohcNXbW/johLhNBCajpwl7Q= 2023-09-29T22:14:19.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oNqPohcNXbW/johLhNBCajpwl7Q=', 'Content-Length': '0'} 2023-09-29T22:14:19.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000567d428e1945465b-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:19.050 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.050 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.050 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.050 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.050 DEBUG:boto:Method: GET 2023-09-29T22:14:19.050 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.051 DEBUG:boto:Data: 2023-09-29T22:14:19.051 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.051 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.051 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.051 DEBUG:boto:Params: {} 2023-09-29T22:14:19.051 DEBUG:boto:Token: None 2023-09-29T22:14:19.051 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.051 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ= 2023-09-29T22:14:19.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ=', 'Content-Length': '0'} 2023-09-29T22:14:19.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cb68f36780f0ad3-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.053 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.053 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.054 DEBUG:boto:path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.054 DEBUG:boto:auth_path=/dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.054 DEBUG:boto:Method: GET 2023-09-29T22:14:19.054 DEBUG:boto:Path: /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.054 DEBUG:boto:Data: 2023-09-29T22:14:19.054 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.054 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.054 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.054 DEBUG:boto:Params: {} 2023-09-29T22:14:19.055 DEBUG:boto:Token: None 2023-09-29T22:14:19.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj?versionId=null 2023-09-29T22:14:19.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ= 2023-09-29T22:14:19.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O5hn6lRyaHiL70lhNhxQl7GaTxQ=', 'Content-Length': '0'} 2023-09-29T22:14:19.056 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd7370a95614ef6-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.057 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.057 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.057 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.057 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.057 DEBUG:boto:Method: GET 2023-09-29T22:14:19.057 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.057 DEBUG:boto:Data: 2023-09-29T22:14:19.057 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.058 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.058 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.058 DEBUG:boto:Params: {} 2023-09-29T22:14:19.058 DEBUG:boto:Token: None 2023-09-29T22:14:19.058 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE= 2023-09-29T22:14:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE=', 'Content-Length': '0'} 2023-09-29T22:14:19.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e574df302201d15f-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.060 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.060 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.061 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.061 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.061 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.061 DEBUG:boto:Method: GET 2023-09-29T22:14:19.061 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.061 DEBUG:boto:Data: 2023-09-29T22:14:19.061 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.061 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.061 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.061 DEBUG:boto:Params: {} 2023-09-29T22:14:19.062 DEBUG:boto:Token: None 2023-09-29T22:14:19.062 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.062 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE= 2023-09-29T22:14:19.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE=', 'Content-Length': '0'} 2023-09-29T22:14:19.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5cf8e21c3bd4382-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.064 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.064 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.065 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.065 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.065 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:19.065 DEBUG:boto:Data: 2023-09-29T22:14:19.065 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.065 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.065 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.065 DEBUG:boto:Params: {} 2023-09-29T22:14:19.065 DEBUG:boto:Token: None 2023-09-29T22:14:19.066 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj 2023-09-29T22:14:19.066 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UQXdIctq9b8le2NlRX7kOB4Vqcg= 2023-09-29T22:14:19.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UQXdIctq9b8le2NlRX7kOB4Vqcg=', 'Content-Length': '0'} 2023-09-29T22:14:19.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d996ea811fa37d9d-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.067 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.068 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.068 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.068 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:19.068 DEBUG:boto:Data: 2023-09-29T22:14:19.068 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.068 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.068 DEBUG:boto:Params: {} 2023-09-29T22:14:19.068 DEBUG:boto:Token: None 2023-09-29T22:14:19.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj 2023-09-29T22:14:19.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UQXdIctq9b8le2NlRX7kOB4Vqcg= 2023-09-29T22:14:19.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UQXdIctq9b8le2NlRX7kOB4Vqcg=', 'Content-Length': '0'} 2023-09-29T22:14:19.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e4e067ecfe19081-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T22:14:19.071 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.071 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.071 DEBUG:boto:Method: GET 2023-09-29T22:14:19.071 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:19.071 DEBUG:boto:Data: 2023-09-29T22:14:19.071 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.072 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.072 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.072 DEBUG:boto:Params: {} 2023-09-29T22:14:19.072 DEBUG:boto:Token: None 2023-09-29T22:14:19.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj 2023-09-29T22:14:19.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jwp54Uz5/BhjWCXzsXnyP1J2vLM= 2023-09-29T22:14:19.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jwp54Uz5/BhjWCXzsXnyP1J2vLM=', 'Content-Length': '0'} 2023-09-29T22:14:19.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e484be917be74718-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.075 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:19.075 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:19.075 DEBUG:boto:Method: GET 2023-09-29T22:14:19.075 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:19.075 DEBUG:boto:Data: 2023-09-29T22:14:19.075 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.075 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.075 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.076 DEBUG:boto:Params: {} 2023-09-29T22:14:19.076 DEBUG:boto:Token: None 2023-09-29T22:14:19.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/myobj 2023-09-29T22:14:19.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jwp54Uz5/BhjWCXzsXnyP1J2vLM= 2023-09-29T22:14:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jwp54Uz5/BhjWCXzsXnyP1J2vLM=', 'Content-Length': '0'} 2023-09-29T22:14:19.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000708ec9ba9a827b1b-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.078 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.078 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.078 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.078 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.078 DEBUG:boto:Method: GET 2023-09-29T22:14:19.078 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.078 DEBUG:boto:Data: 2023-09-29T22:14:19.079 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.079 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:19.079 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.079 DEBUG:boto:Params: {} 2023-09-29T22:14:19.079 DEBUG:boto:Token: None 2023-09-29T22:14:19.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.079 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE= 2023-09-29T22:14:19.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE=', 'Content-Length': '0'} 2023-09-29T22:14:19.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaede98527474bf6-0065174c3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.082 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.082 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:19.082 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:19.082 DEBUG:boto:path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.082 DEBUG:boto:auth_path=/dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.083 DEBUG:boto:Method: GET 2023-09-29T22:14:19.083 DEBUG:boto:Path: /dmuhsu-15/?prefix=myobj 2023-09-29T22:14:19.083 DEBUG:boto:Data: 2023-09-29T22:14:19.083 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.083 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:19.083 DEBUG:boto:Port: 8001 2023-09-29T22:14:19.083 DEBUG:boto:Params: {} 2023-09-29T22:14:19.083 DEBUG:boto:Token: None 2023-09-29T22:14:19.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-15/ 2023-09-29T22:14:19.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE= 2023-09-29T22:14:19.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZV5B9eeHijJwLxrtMu3YZIfsPNE=', 'Content-Length': '0'} 2023-09-29T22:14:19.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c23f875aa8ba9a1-0065174c3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.086 DEBUG:boto:b'dmuhsu-15myobj1000falsemyobj2023-09-29T22:12:37.320Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:14:19.086 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-15 zones={test-zone2, test-zone3} 2023-09-29T22:14:19.086 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:19.086 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:19.086 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.087 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:19.087 DEBUG:boto:Data: 2023-09-29T22:14:19.087 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.087 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:19.087 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.087 DEBUG:boto:Params: {} 2023-09-29T22:14:19.087 DEBUG:boto:Token: None 2023-09-29T22:14:19.087 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-13/ 2023-09-29T22:14:19.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JM2JLfBKeqXYv3wMqrM1YcotC0s= 2023-09-29T22:14:19.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JM2JLfBKeqXYv3wMqrM1YcotC0s=', 'Content-Length': '0'} 2023-09-29T22:14:19.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d855efa0027992a4-0065174c3b-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.090 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:19.090 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:19.090 DEBUG:boto:Method: HEAD 2023-09-29T22:14:19.090 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:19.090 DEBUG:boto:Data: 2023-09-29T22:14:19.091 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.091 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:19.091 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.091 DEBUG:boto:Params: {} 2023-09-29T22:14:19.091 DEBUG:boto:Token: None 2023-09-29T22:14:19.091 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-13/myobj 2023-09-29T22:14:19.091 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7IybbAmZCuLaMRaP6jJP5V8h1/A= 2023-09-29T22:14:19.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7IybbAmZCuLaMRaP6jJP5V8h1/A=', 'Content-Length': '0'} 2023-09-29T22:14:19.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef24ae92e5f376ed-0065174c3b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.093 DEBUG:boto:path=/dmuhsu-13/myobj 2023-09-29T22:14:19.093 DEBUG:boto:auth_path=/dmuhsu-13/myobj 2023-09-29T22:14:19.093 DEBUG:boto:Method: DELETE 2023-09-29T22:14:19.093 DEBUG:boto:Path: /dmuhsu-13/myobj 2023-09-29T22:14:19.094 DEBUG:boto:Data: 2023-09-29T22:14:19.094 DEBUG:boto:Headers: {} 2023-09-29T22:14:19.094 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:19.094 DEBUG:boto:Port: 8000 2023-09-29T22:14:19.094 DEBUG:boto:Params: {} 2023-09-29T22:14:19.094 DEBUG:boto:Token: None 2023-09-29T22:14:19.095 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:14:19 GMT /dmuhsu-13/myobj 2023-09-29T22:14:19.095 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gvEEgzmGFcXlKG2yk6B3WPExZQw= 2023-09-29T22:14:19.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gvEEgzmGFcXlKG2yk6B3WPExZQw=', 'Content-Length': '0'} 2023-09-29T22:14:19.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d531f0662e9a7e2-0065174c3b-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:19.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:19.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:19.101 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:19.154 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:19.154 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:19.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:19.186+0000 7fa879212880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:19.187 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:19.187 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:24.187 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:24.187+0000 7fa879212880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:24.187 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:24.188 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:29.188+0000 7fa879212880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:29.189 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:29.189 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:34.190 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:34.189+0000 7fa879212880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:34.190 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.14.6, , , , , , , , , ] 2023-09-29T22:14:34.190 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:39.191 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:39.191+0000 7fa879212880 1 bucket sync caught up with source: 2023-09-29T22:14:39.192 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:39.192 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:39.192 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:14:39.191+0000 7fa879212880 0 bucket checkpoint complete 2023-09-29T22:14:39.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-13 zones={test-zone2, test-zone1} 2023-09-29T22:14:39.207 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.207 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.207 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.208 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:39.208 DEBUG:boto:Data: 2023-09-29T22:14:39.208 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.208 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.208 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.208 DEBUG:boto:Params: {} 2023-09-29T22:14:39.209 DEBUG:boto:Token: None 2023-09-29T22:14:39.209 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/ 2023-09-29T22:14:39.209 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM= 2023-09-29T22:14:39.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM=', 'Content-Length': '0'} 2023-09-29T22:14:39.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbf88c27c5946daf-0065174c4f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.212 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.212 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.213 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.213 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:39.213 DEBUG:boto:Data: 2023-09-29T22:14:39.213 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.213 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.213 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.213 DEBUG:boto:Params: {} 2023-09-29T22:14:39.213 DEBUG:boto:Token: None 2023-09-29T22:14:39.214 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/ 2023-09-29T22:14:39.214 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM= 2023-09-29T22:14:39.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM=', 'Content-Length': '0'} 2023-09-29T22:14:39.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077fc8073e4ba40d9-0065174c4f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.216 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:39.217 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.217 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.217 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.217 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.217 DEBUG:boto:Method: GET 2023-09-29T22:14:39.217 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.217 DEBUG:boto:Data: 2023-09-29T22:14:39.217 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.217 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.218 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.218 DEBUG:boto:Params: {} 2023-09-29T22:14:39.218 DEBUG:boto:Token: None 2023-09-29T22:14:39.218 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.218 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001941f5b40dcc7d96-0065174c4f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.221 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.221 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:39.221 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.221 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.221 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.222 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.222 DEBUG:boto:Method: GET 2023-09-29T22:14:39.222 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.222 DEBUG:boto:Data: 2023-09-29T22:14:39.222 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.222 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.222 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.222 DEBUG:boto:Params: {} 2023-09-29T22:14:39.222 DEBUG:boto:Token: None 2023-09-29T22:14:39.223 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd944a44aee29d30-0065174c4f-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.225 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.226 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.226 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.226 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.226 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.226 DEBUG:boto:Method: GET 2023-09-29T22:14:39.226 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.226 DEBUG:boto:Data: 2023-09-29T22:14:39.227 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.227 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.227 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.227 DEBUG:boto:Params: {} 2023-09-29T22:14:39.227 DEBUG:boto:Token: None 2023-09-29T22:14:39.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.227 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a26346afbde33fd-0065174c4f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.229 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.230 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.230 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.230 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.230 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.230 DEBUG:boto:Method: GET 2023-09-29T22:14:39.230 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.230 DEBUG:boto:Data: 2023-09-29T22:14:39.230 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.231 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.231 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.231 DEBUG:boto:Params: {} 2023-09-29T22:14:39.231 DEBUG:boto:Token: None 2023-09-29T22:14:39.231 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.231 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b6f05e83e280cc-0065174c4f-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.233 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-13 zones={test-zone2, test-zone1} 2023-09-29T22:14:39.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:39.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:39.234 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:39.292 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:39.293 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:39.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:39.329+0000 7fcd59536880 1 bucket sync caught up with source: 2023-09-29T22:14:39.329 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:39.329 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-09-29T22:14:39.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:39.329+0000 7fcd59536880 0 bucket checkpoint complete 2023-09-29T22:14:39.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-13 zones={test-zone3, test-zone1} 2023-09-29T22:14:39.341 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.341 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.341 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.341 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:39.341 DEBUG:boto:Data: 2023-09-29T22:14:39.341 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.341 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:39.342 DEBUG:boto:Port: 8001 2023-09-29T22:14:39.342 DEBUG:boto:Params: {} 2023-09-29T22:14:39.342 DEBUG:boto:Token: None 2023-09-29T22:14:39.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/ 2023-09-29T22:14:39.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM= 2023-09-29T22:14:39.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM=', 'Content-Length': '0'} 2023-09-29T22:14:39.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c45b4b717ccbe8-0065174c4f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.345 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.345 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.345 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.345 DEBUG:boto:Path: /dmuhsu-13/ 2023-09-29T22:14:39.345 DEBUG:boto:Data: 2023-09-29T22:14:39.345 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.345 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.345 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.346 DEBUG:boto:Params: {} 2023-09-29T22:14:39.346 DEBUG:boto:Token: None 2023-09-29T22:14:39.346 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/ 2023-09-29T22:14:39.346 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM= 2023-09-29T22:14:39.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qLcyVHQE4jv0FTZ5bB0A89HZXNM=', 'Content-Length': '0'} 2023-09-29T22:14:39.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004583265d348bfbc7-0065174c4f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:39.349 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.349 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.349 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.349 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.349 DEBUG:boto:Method: GET 2023-09-29T22:14:39.349 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.349 DEBUG:boto:Data: 2023-09-29T22:14:39.349 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.349 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:39.349 DEBUG:boto:Port: 8001 2023-09-29T22:14:39.349 DEBUG:boto:Params: {} 2023-09-29T22:14:39.350 DEBUG:boto:Token: None 2023-09-29T22:14:39.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abdefc9249b675af-0065174c4f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.352 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.352 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:39.353 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.353 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.353 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.353 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.353 DEBUG:boto:Method: GET 2023-09-29T22:14:39.353 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.353 DEBUG:boto:Data: 2023-09-29T22:14:39.353 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.354 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.354 DEBUG:boto:Params: {} 2023-09-29T22:14:39.354 DEBUG:boto:Token: None 2023-09-29T22:14:39.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.354 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058195987f025da46-0065174c4f-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.356 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.357 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.357 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.357 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.357 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.357 DEBUG:boto:Method: GET 2023-09-29T22:14:39.357 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.357 DEBUG:boto:Data: 2023-09-29T22:14:39.357 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.358 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:39.358 DEBUG:boto:Port: 8001 2023-09-29T22:14:39.358 DEBUG:boto:Params: {} 2023-09-29T22:14:39.358 DEBUG:boto:Token: None 2023-09-29T22:14:39.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.358 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b67c807075ccb60-0065174c4f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.360 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.361 DEBUG:boto:path=/dmuhsu-13/ 2023-09-29T22:14:39.361 DEBUG:boto:auth_path=/dmuhsu-13/ 2023-09-29T22:14:39.361 DEBUG:boto:path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.361 DEBUG:boto:auth_path=/dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.361 DEBUG:boto:Method: GET 2023-09-29T22:14:39.361 DEBUG:boto:Path: /dmuhsu-13/?versions&max-keys=999 2023-09-29T22:14:39.361 DEBUG:boto:Data: 2023-09-29T22:14:39.362 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.362 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:39.362 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.362 DEBUG:boto:Params: {} 2023-09-29T22:14:39.362 DEBUG:boto:Token: None 2023-09-29T22:14:39.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-13/?versions 2023-09-29T22:14:39.362 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk= 2023-09-29T22:14:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c7oGJysTG0sx5TItQYHk/Zi+sGk=', 'Content-Length': '0'} 2023-09-29T22:14:39.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723316572734b787-0065174c4f-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.364 DEBUG:boto:b'dmuhsu-13999false' 2023-09-29T22:14:39.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-13 zones={test-zone3, test-zone1} 2023-09-29T22:14:39.365 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:39.366 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:39.366 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.366 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:39.366 DEBUG:boto:Data: 2023-09-29T22:14:39.366 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.366 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.366 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.366 DEBUG:boto:Params: {} 2023-09-29T22:14:39.367 DEBUG:boto:Token: None 2023-09-29T22:14:39.367 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-14/ 2023-09-29T22:14:39.367 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VDTBPMmeRDqHiBgPjWPWh2keBsk= 2023-09-29T22:14:39.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VDTBPMmeRDqHiBgPjWPWh2keBsk=', 'Content-Length': '0'} 2023-09-29T22:14:39.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057f80df0dcd7bb7e-0065174c4f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.370 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:39.370 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:39.370 DEBUG:boto:Method: HEAD 2023-09-29T22:14:39.370 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:39.370 DEBUG:boto:Data: 2023-09-29T22:14:39.370 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.370 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.370 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.371 DEBUG:boto:Params: {} 2023-09-29T22:14:39.371 DEBUG:boto:Token: None 2023-09-29T22:14:39.371 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-14/myobj 2023-09-29T22:14:39.371 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KGTaRb1rsWb6a2RMY3TKuNe0DeU= 2023-09-29T22:14:39.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KGTaRb1rsWb6a2RMY3TKuNe0DeU=', 'Content-Length': '0'} 2023-09-29T22:14:39.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a65c868cef9bc397-0065174c4f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.375 DEBUG:boto:path=/dmuhsu-14/myobj 2023-09-29T22:14:39.375 DEBUG:boto:auth_path=/dmuhsu-14/myobj 2023-09-29T22:14:39.375 DEBUG:boto:Method: DELETE 2023-09-29T22:14:39.375 DEBUG:boto:Path: /dmuhsu-14/myobj 2023-09-29T22:14:39.376 DEBUG:boto:Data: 2023-09-29T22:14:39.376 DEBUG:boto:Headers: {} 2023-09-29T22:14:39.376 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:39.376 DEBUG:boto:Port: 8000 2023-09-29T22:14:39.376 DEBUG:boto:Params: {} 2023-09-29T22:14:39.376 DEBUG:boto:Token: None 2023-09-29T22:14:39.376 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:14:39 GMT /dmuhsu-14/myobj 2023-09-29T22:14:39.376 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QzD5y1wz1oTKJREUZzWcGMmW1R4= 2023-09-29T22:14:39.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QzD5y1wz1oTKJREUZzWcGMmW1R4=', 'Content-Length': '0'} 2023-09-29T22:14:39.382 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa4140e4bb15c407-0065174c4f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:14:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:39.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:39.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:39.383 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:39.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:39.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:39.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:39.468+0000 7f3631667880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:39.468 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:39.469 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:44.470+0000 7f3631667880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:44.470 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:44.470 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:49.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:49.470+0000 7f3631667880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:49.471 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:49.471 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:54.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:54.472+0000 7f3631667880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:54.473 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:54.473 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:59.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:59.473+0000 7f3631667880 1 bucket sync caught up with source: 2023-09-29T22:14:59.474 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:59.474 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:59.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:59.473+0000 7f3631667880 0 bucket checkpoint complete 2023-09-29T22:14:59.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-14 zones={test-zone1, test-zone2} 2023-09-29T22:14:59.490 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.490 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.490 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.490 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:59.490 DEBUG:boto:Data: 2023-09-29T22:14:59.490 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.490 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:59.490 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.491 DEBUG:boto:Params: {} 2023-09-29T22:14:59.491 DEBUG:boto:Token: None 2023-09-29T22:14:59.491 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/ 2023-09-29T22:14:59.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA= 2023-09-29T22:14:59.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA=', 'Content-Length': '0'} 2023-09-29T22:14:59.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a9648b656a82e3d-0065174c63-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.494 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.494 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.494 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.494 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:59.494 DEBUG:boto:Data: 2023-09-29T22:14:59.494 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.495 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.495 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.495 DEBUG:boto:Params: {} 2023-09-29T22:14:59.495 DEBUG:boto:Token: None 2023-09-29T22:14:59.495 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/ 2023-09-29T22:14:59.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA= 2023-09-29T22:14:59.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA=', 'Content-Length': '0'} 2023-09-29T22:14:59.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000214532dd8708b1f3-0065174c63-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:59.498 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.498 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.498 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.498 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.499 DEBUG:boto:Method: GET 2023-09-29T22:14:59.499 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.499 DEBUG:boto:Data: 2023-09-29T22:14:59.499 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.499 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:59.499 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.499 DEBUG:boto:Params: {} 2023-09-29T22:14:59.499 DEBUG:boto:Token: None 2023-09-29T22:14:59.499 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9a5a34acc78f42-0065174c63-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.502 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:59.503 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.503 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.503 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.503 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.503 DEBUG:boto:Method: GET 2023-09-29T22:14:59.503 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.503 DEBUG:boto:Data: 2023-09-29T22:14:59.503 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.503 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.504 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.504 DEBUG:boto:Params: {} 2023-09-29T22:14:59.504 DEBUG:boto:Token: None 2023-09-29T22:14:59.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004876629186aa437b-0065174c63-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.506 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.507 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.507 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.507 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.507 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.507 DEBUG:boto:Method: GET 2023-09-29T22:14:59.507 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.507 DEBUG:boto:Data: 2023-09-29T22:14:59.507 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.508 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:14:59.508 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.508 DEBUG:boto:Params: {} 2023-09-29T22:14:59.508 DEBUG:boto:Token: None 2023-09-29T22:14:59.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644efe4aeaf7502a-0065174c63-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.511 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.511 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.511 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.511 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.511 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.512 DEBUG:boto:Method: GET 2023-09-29T22:14:59.512 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.512 DEBUG:boto:Data: 2023-09-29T22:14:59.512 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.512 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.512 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.512 DEBUG:boto:Params: {} 2023-09-29T22:14:59.512 DEBUG:boto:Token: None 2023-09-29T22:14:59.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c70d98a1ff0c617e-0065174c63-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.515 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.515 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-14 zones={test-zone1, test-zone2} 2023-09-29T22:14:59.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:59.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:59.516 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:59.573 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:59.574 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:59.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:59.609+0000 7efe0a212880 1 bucket sync caught up with source: 2023-09-29T22:14:59.610 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:59.610 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.27.6, , , , , , , , , ] 2023-09-29T22:14:59.610 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:59.609+0000 7efe0a212880 0 bucket checkpoint complete 2023-09-29T22:14:59.622 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-14 zones={test-zone3, test-zone2} 2023-09-29T22:14:59.622 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.622 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.622 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.622 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:59.622 DEBUG:boto:Data: 2023-09-29T22:14:59.622 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.622 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.623 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.623 DEBUG:boto:Params: {} 2023-09-29T22:14:59.623 DEBUG:boto:Token: None 2023-09-29T22:14:59.623 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/ 2023-09-29T22:14:59.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA= 2023-09-29T22:14:59.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA=', 'Content-Length': '0'} 2023-09-29T22:14:59.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000455f43a24d26c094-0065174c63-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.626 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.626 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.626 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.627 DEBUG:boto:Path: /dmuhsu-14/ 2023-09-29T22:14:59.627 DEBUG:boto:Data: 2023-09-29T22:14:59.627 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.627 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.627 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.627 DEBUG:boto:Params: {} 2023-09-29T22:14:59.627 DEBUG:boto:Token: None 2023-09-29T22:14:59.627 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/ 2023-09-29T22:14:59.627 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA= 2023-09-29T22:14:59.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E0GtPIJCpgZpu9yuXDdrFWJLbdA=', 'Content-Length': '0'} 2023-09-29T22:14:59.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007512135697209ad4-0065174c63-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.632 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:14:59.632 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.632 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.632 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.632 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.632 DEBUG:boto:Method: GET 2023-09-29T22:14:59.632 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.632 DEBUG:boto:Data: 2023-09-29T22:14:59.632 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.633 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.633 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.633 DEBUG:boto:Params: {} 2023-09-29T22:14:59.633 DEBUG:boto:Token: None 2023-09-29T22:14:59.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.634 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6b7fcaa5793506-0065174c63-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.636 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:14:59.636 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.637 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.637 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.637 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.637 DEBUG:boto:Method: GET 2023-09-29T22:14:59.637 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.637 DEBUG:boto:Data: 2023-09-29T22:14:59.637 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.637 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.637 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.637 DEBUG:boto:Params: {} 2023-09-29T22:14:59.637 DEBUG:boto:Token: None 2023-09-29T22:14:59.638 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.638 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7a37ad83d221c3b-0065174c63-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.640 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.640 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.641 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.641 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.641 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.641 DEBUG:boto:Method: GET 2023-09-29T22:14:59.641 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.641 DEBUG:boto:Data: 2023-09-29T22:14:59.641 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.641 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.641 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.641 DEBUG:boto:Params: {} 2023-09-29T22:14:59.642 DEBUG:boto:Token: None 2023-09-29T22:14:59.642 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.642 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c4b59431c9909c-0065174c63-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.644 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.645 DEBUG:boto:path=/dmuhsu-14/ 2023-09-29T22:14:59.645 DEBUG:boto:auth_path=/dmuhsu-14/ 2023-09-29T22:14:59.645 DEBUG:boto:path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.645 DEBUG:boto:auth_path=/dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.645 DEBUG:boto:Method: GET 2023-09-29T22:14:59.645 DEBUG:boto:Path: /dmuhsu-14/?versions&max-keys=999 2023-09-29T22:14:59.645 DEBUG:boto:Data: 2023-09-29T22:14:59.645 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.646 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:14:59.646 DEBUG:boto:Port: 8000 2023-09-29T22:14:59.646 DEBUG:boto:Params: {} 2023-09-29T22:14:59.646 DEBUG:boto:Token: None 2023-09-29T22:14:59.646 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-14/?versions 2023-09-29T22:14:59.646 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E= 2023-09-29T22:14:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vu8wpc96SDY5lHk207D9lEaAn0E=', 'Content-Length': '0'} 2023-09-29T22:14:59.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651d764ece100e26-0065174c63-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.649 DEBUG:boto:b'dmuhsu-14999false' 2023-09-29T22:14:59.649 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-14 zones={test-zone3, test-zone2} 2023-09-29T22:14:59.649 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:14:59.649 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:14:59.650 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.650 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:14:59.650 DEBUG:boto:Data: 2023-09-29T22:14:59.650 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.650 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.650 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.650 DEBUG:boto:Params: {} 2023-09-29T22:14:59.650 DEBUG:boto:Token: None 2023-09-29T22:14:59.651 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-15/ 2023-09-29T22:14:59.651 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F4Q8EC2+Zy9NHYxWUDtrNbuO7Ls= 2023-09-29T22:14:59.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F4Q8EC2+Zy9NHYxWUDtrNbuO7Ls=', 'Content-Length': '0'} 2023-09-29T22:14:59.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ac24a1e88175500-0065174c63-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.653 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:59.653 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:59.653 DEBUG:boto:Method: HEAD 2023-09-29T22:14:59.653 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:59.653 DEBUG:boto:Data: 2023-09-29T22:14:59.653 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.653 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.653 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.653 DEBUG:boto:Params: {} 2023-09-29T22:14:59.654 DEBUG:boto:Token: None 2023-09-29T22:14:59.654 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-15/myobj 2023-09-29T22:14:59.654 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l/t62KlcmclOdgXuknJdSDsawXY= 2023-09-29T22:14:59.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l/t62KlcmclOdgXuknJdSDsawXY=', 'Content-Length': '0'} 2023-09-29T22:14:59.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:12:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000966fcaa6b1066f20-0065174c63-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.655 DEBUG:boto:path=/dmuhsu-15/myobj 2023-09-29T22:14:59.656 DEBUG:boto:auth_path=/dmuhsu-15/myobj 2023-09-29T22:14:59.656 DEBUG:boto:Method: DELETE 2023-09-29T22:14:59.656 DEBUG:boto:Path: /dmuhsu-15/myobj 2023-09-29T22:14:59.656 DEBUG:boto:Data: 2023-09-29T22:14:59.656 DEBUG:boto:Headers: {} 2023-09-29T22:14:59.656 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:14:59.656 DEBUG:boto:Port: 8001 2023-09-29T22:14:59.656 DEBUG:boto:Params: {} 2023-09-29T22:14:59.656 DEBUG:boto:Token: None 2023-09-29T22:14:59.656 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:14:59 GMT /dmuhsu-15/myobj 2023-09-29T22:14:59.657 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eFfy9mxzPOm+OciFcr8Ns33evM4= 2023-09-29T22:14:59.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:14:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eFfy9mxzPOm+OciFcr8Ns33evM4=', 'Content-Length': '0'} 2023-09-29T22:14:59.661 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007fee4577e86e4827-0065174c63-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:14:59.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:59.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:14:59.662 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:14:59.714 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:59.714 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:59.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:14:59.745+0000 7fba5d044880 1 waiting for incremental sync to catch up: 2023-09-29T22:14:59.746 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:14:59.746 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:04.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:04.746+0000 7fba5d044880 1 waiting for incremental sync to catch up: 2023-09-29T22:15:04.747 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:15:04.747 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:09.748 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:09.749+0000 7fba5d044880 1 waiting for incremental sync to catch up: 2023-09-29T22:15:09.749 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:15:09.749 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:14.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:14.750+0000 7fba5d044880 1 waiting for incremental sync to catch up: 2023-09-29T22:15:14.750 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2023-09-29T22:15:14.750 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:19.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:19.751+0000 7fba5d044880 1 bucket sync caught up with source: 2023-09-29T22:15:19.751 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:19.751 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:19.751 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:19.751+0000 7fba5d044880 0 bucket checkpoint complete 2023-09-29T22:15:19.766 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-15 zones={test-zone1, test-zone3} 2023-09-29T22:15:19.767 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.767 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.767 DEBUG:boto:Method: HEAD 2023-09-29T22:15:19.767 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:15:19.767 DEBUG:boto:Data: 2023-09-29T22:15:19.768 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.768 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:19.768 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.768 DEBUG:boto:Params: {} 2023-09-29T22:15:19.769 DEBUG:boto:Token: None 2023-09-29T22:15:19.769 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/ 2023-09-29T22:15:19.769 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII= 2023-09-29T22:15:19.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII=', 'Content-Length': '0'} 2023-09-29T22:15:19.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852522ef7d9efbd8-0065174c77-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.772 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.772 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.773 DEBUG:boto:Method: HEAD 2023-09-29T22:15:19.773 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:15:19.773 DEBUG:boto:Data: 2023-09-29T22:15:19.773 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.773 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.773 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.774 DEBUG:boto:Params: {} 2023-09-29T22:15:19.774 DEBUG:boto:Token: None 2023-09-29T22:15:19.774 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/ 2023-09-29T22:15:19.774 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII= 2023-09-29T22:15:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII=', 'Content-Length': '0'} 2023-09-29T22:15:19.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a307bada18b919-0065174c77-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.777 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:15:19.777 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.778 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.778 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.778 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.778 DEBUG:boto:Method: GET 2023-09-29T22:15:19.778 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.779 DEBUG:boto:Data: 2023-09-29T22:15:19.779 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.779 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:19.779 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.779 DEBUG:boto:Params: {} 2023-09-29T22:15:19.780 DEBUG:boto:Token: None 2023-09-29T22:15:19.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d596a9bed66d39c9-0065174c77-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.783 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:15:19.784 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.784 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.784 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.784 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.784 DEBUG:boto:Method: GET 2023-09-29T22:15:19.784 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.785 DEBUG:boto:Data: 2023-09-29T22:15:19.785 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.785 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.785 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.785 DEBUG:boto:Params: {} 2023-09-29T22:15:19.785 DEBUG:boto:Token: None 2023-09-29T22:15:19.786 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.786 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004976ac60f684c8fd-0065174c77-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.789 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.789 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.790 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.790 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.790 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.790 DEBUG:boto:Method: GET 2023-09-29T22:15:19.790 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.790 DEBUG:boto:Data: 2023-09-29T22:15:19.790 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.790 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:19.790 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.790 DEBUG:boto:Params: {} 2023-09-29T22:15:19.791 DEBUG:boto:Token: None 2023-09-29T22:15:19.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.791 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a91ce11c298d20bf-0065174c77-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.834 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.835 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.835 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.835 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.835 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.835 DEBUG:boto:Method: GET 2023-09-29T22:15:19.836 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.836 DEBUG:boto:Data: 2023-09-29T22:15:19.836 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.836 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.836 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.836 DEBUG:boto:Params: {} 2023-09-29T22:15:19.837 DEBUG:boto:Token: None 2023-09-29T22:15:19.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.837 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e21d8040754a94-0065174c77-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.840 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.840 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-15 zones={test-zone1, test-zone3} 2023-09-29T22:15:19.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:19.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:19.841 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:19.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:19.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:19.939 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:19.938+0000 7fd924a67880 1 bucket sync caught up with source: 2023-09-29T22:15:19.939 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:19.939 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.22.6, , , , , , , , , ] 2023-09-29T22:15:19.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:19.938+0000 7fd924a67880 0 bucket checkpoint complete 2023-09-29T22:15:19.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-15 zones={test-zone2, test-zone3} 2023-09-29T22:15:19.953 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.954 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.954 DEBUG:boto:Method: HEAD 2023-09-29T22:15:19.954 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:15:19.954 DEBUG:boto:Data: 2023-09-29T22:15:19.954 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.954 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:19.955 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.955 DEBUG:boto:Params: {} 2023-09-29T22:15:19.955 DEBUG:boto:Token: None 2023-09-29T22:15:19.955 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/ 2023-09-29T22:15:19.955 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII= 2023-09-29T22:15:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII=', 'Content-Length': '0'} 2023-09-29T22:15:19.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006715f623094ec119-0065174c77-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.958 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.958 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.958 DEBUG:boto:Method: HEAD 2023-09-29T22:15:19.959 DEBUG:boto:Path: /dmuhsu-15/ 2023-09-29T22:15:19.959 DEBUG:boto:Data: 2023-09-29T22:15:19.959 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.959 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.959 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.959 DEBUG:boto:Params: {} 2023-09-29T22:15:19.959 DEBUG:boto:Token: None 2023-09-29T22:15:19.959 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/ 2023-09-29T22:15:19.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII= 2023-09-29T22:15:19.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fEwRuAezOLB0zg1I92WkcbscQII=', 'Content-Length': '0'} 2023-09-29T22:15:19.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de4179f2c0580db5-0065174c77-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:15:19.962 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.962 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.962 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.963 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.963 DEBUG:boto:Method: GET 2023-09-29T22:15:19.963 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.963 DEBUG:boto:Data: 2023-09-29T22:15:19.963 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.963 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:19.963 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.963 DEBUG:boto:Params: {} 2023-09-29T22:15:19.964 DEBUG:boto:Token: None 2023-09-29T22:15:19.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c70fe5212cdb7b36-0065174c77-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.966 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:15:19.967 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.967 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.967 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.967 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.967 DEBUG:boto:Method: GET 2023-09-29T22:15:19.967 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.967 DEBUG:boto:Data: 2023-09-29T22:15:19.967 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.968 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.968 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.968 DEBUG:boto:Params: {} 2023-09-29T22:15:19.968 DEBUG:boto:Token: None 2023-09-29T22:15:19.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae19289b3346ae5-0065174c77-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.971 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.971 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.971 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.971 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.971 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.972 DEBUG:boto:Method: GET 2023-09-29T22:15:19.972 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.972 DEBUG:boto:Data: 2023-09-29T22:15:19.972 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.972 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:19.972 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.972 DEBUG:boto:Params: {} 2023-09-29T22:15:19.972 DEBUG:boto:Token: None 2023-09-29T22:15:19.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76d4dd69470a08d-0065174c77-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.974 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.975 DEBUG:boto:path=/dmuhsu-15/ 2023-09-29T22:15:19.975 DEBUG:boto:auth_path=/dmuhsu-15/ 2023-09-29T22:15:19.975 DEBUG:boto:path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.975 DEBUG:boto:auth_path=/dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.975 DEBUG:boto:Method: GET 2023-09-29T22:15:19.975 DEBUG:boto:Path: /dmuhsu-15/?versions&max-keys=999 2023-09-29T22:15:19.975 DEBUG:boto:Data: 2023-09-29T22:15:19.975 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:19.976 DEBUG:boto:Port: 8001 2023-09-29T22:15:19.976 DEBUG:boto:Params: {} 2023-09-29T22:15:19.976 DEBUG:boto:Token: None 2023-09-29T22:15:19.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-15/?versions 2023-09-29T22:15:19.976 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8= 2023-09-29T22:15:19.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:26zvREDWCTYNiLfH68EBzVbmEu8=', 'Content-Length': '0'} 2023-09-29T22:15:19.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fba119e82f45ed-0065174c77-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.979 DEBUG:boto:b'dmuhsu-15999false' 2023-09-29T22:15:19.979 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-15 zones={test-zone2, test-zone3} 2023-09-29T22:15:19.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-09-29T22:15:19.980 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-16 2023-09-29T22:15:19.980 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:15:19.980 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:15:19.981 DEBUG:boto:Method: PUT 2023-09-29T22:15:19.981 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:15:19.981 DEBUG:boto:Data: 2023-09-29T22:15:19.981 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:19.981 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.981 DEBUG:boto:Params: {} 2023-09-29T22:15:19.981 DEBUG:boto:Token: None 2023-09-29T22:15:19.981 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-16/ 2023-09-29T22:15:19.981 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gfJrVUVPq+BSDOzyYOE+ZgCqxKo= 2023-09-29T22:15:19.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gfJrVUVPq+BSDOzyYOE+ZgCqxKo=', 'Content-Length': '0'} 2023-09-29T22:15:19.997 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000484855e1e8ded74-0065174c77-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:19.997 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-17 2023-09-29T22:15:19.997 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:15:19.997 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:15:19.997 DEBUG:boto:Method: PUT 2023-09-29T22:15:19.997 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:15:19.997 DEBUG:boto:Data: 2023-09-29T22:15:19.997 DEBUG:boto:Headers: {} 2023-09-29T22:15:19.997 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:19.998 DEBUG:boto:Port: 8000 2023-09-29T22:15:19.998 DEBUG:boto:Params: {} 2023-09-29T22:15:19.998 DEBUG:boto:Token: None 2023-09-29T22:15:19.998 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:19 GMT /dmuhsu-17/ 2023-09-29T22:15:19.998 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kmZ9xZKBwDjBN6k2GZL/q51m5tg= 2023-09-29T22:15:19.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kmZ9xZKBwDjBN6k2GZL/q51m5tg=', 'Content-Length': '0'} 2023-09-29T22:15:20.029 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8750f28b89c2dd7-0065174c77-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:20.029 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-18 2023-09-29T22:15:20.029 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:15:20.029 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:15:20.029 DEBUG:boto:Method: PUT 2023-09-29T22:15:20.029 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:15:20.030 DEBUG:boto:Data: 2023-09-29T22:15:20.030 DEBUG:boto:Headers: {} 2023-09-29T22:15:20.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:20.030 DEBUG:boto:Port: 8001 2023-09-29T22:15:20.030 DEBUG:boto:Params: {} 2023-09-29T22:15:20.030 DEBUG:boto:Token: None 2023-09-29T22:15:20.030 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:20 GMT /dmuhsu-18/ 2023-09-29T22:15:20.030 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GPF/RvdfwauklJYM+bt7gjgxjt8= 2023-09-29T22:15:20.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GPF/RvdfwauklJYM+bt7gjgxjt8=', 'Content-Length': '0'} 2023-09-29T22:15:20.066 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002137f9c63eff2a9a-0065174c78-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:20.066 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:15:20.066 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:15:20.066 DEBUG:boto:path=/dmuhsu-16/?versioning 2023-09-29T22:15:20.066 DEBUG:boto:auth_path=/dmuhsu-16/?versioning 2023-09-29T22:15:20.066 DEBUG:boto:Method: PUT 2023-09-29T22:15:20.066 DEBUG:boto:Path: /dmuhsu-16/?versioning 2023-09-29T22:15:20.067 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:15:20.067 DEBUG:boto:Headers: {} 2023-09-29T22:15:20.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:20.067 DEBUG:boto:Port: 8000 2023-09-29T22:15:20.067 DEBUG:boto:Params: {} 2023-09-29T22:15:20.067 DEBUG:boto:Token: None 2023-09-29T22:15:20.067 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:20 GMT /dmuhsu-16/?versioning 2023-09-29T22:15:20.067 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XjCTr/r6dJIdhyMqBWeJSRNA00s= 2023-09-29T22:15:20.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XjCTr/r6dJIdhyMqBWeJSRNA00s=', 'Content-Length': '228'} 2023-09-29T22:15:20.072 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ff9ef52e3b87281-0065174c78-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:20.072 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:15:20.072 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:15:20.072 DEBUG:boto:path=/dmuhsu-17/?versioning 2023-09-29T22:15:20.072 DEBUG:boto:auth_path=/dmuhsu-17/?versioning 2023-09-29T22:15:20.072 DEBUG:boto:Method: PUT 2023-09-29T22:15:20.072 DEBUG:boto:Path: /dmuhsu-17/?versioning 2023-09-29T22:15:20.073 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:15:20.073 DEBUG:boto:Headers: {} 2023-09-29T22:15:20.073 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:20.073 DEBUG:boto:Port: 8000 2023-09-29T22:15:20.073 DEBUG:boto:Params: {} 2023-09-29T22:15:20.073 DEBUG:boto:Token: None 2023-09-29T22:15:20.073 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:20 GMT /dmuhsu-17/?versioning 2023-09-29T22:15:20.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LvTb9felYL3hMVkOxaqKNZfqc44= 2023-09-29T22:15:20.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LvTb9felYL3hMVkOxaqKNZfqc44=', 'Content-Length': '228'} 2023-09-29T22:15:20.081 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f106e370326b478-0065174c78-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:20.081 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:15:20.081 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:15:20.081 DEBUG:boto:path=/dmuhsu-18/?versioning 2023-09-29T22:15:20.081 DEBUG:boto:auth_path=/dmuhsu-18/?versioning 2023-09-29T22:15:20.081 DEBUG:boto:Method: PUT 2023-09-29T22:15:20.081 DEBUG:boto:Path: /dmuhsu-18/?versioning 2023-09-29T22:15:20.081 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:15:20.082 DEBUG:boto:Headers: {} 2023-09-29T22:15:20.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:15:20.082 DEBUG:boto:Port: 8001 2023-09-29T22:15:20.082 DEBUG:boto:Params: {} 2023-09-29T22:15:20.082 DEBUG:boto:Token: None 2023-09-29T22:15:20.082 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:20 GMT /dmuhsu-18/?versioning 2023-09-29T22:15:20.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:533DJOmp9bWJy/pxRvjzqU/fths= 2023-09-29T22:15:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:533DJOmp9bWJy/pxRvjzqU/fths=', 'Content-Length': '228'} 2023-09-29T22:15:20.089 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008952ad9500bbc478-0065174c78-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:20.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:20.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:20.090 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:20.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:20.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:20.166 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:15:20.177 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:15:20.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025720.088622_101.1", 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:15:20.088622Z" 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:20.179 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:15:20.180 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:20.181 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:15:20.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:20.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:20.181 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:20.233 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:20.233 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:20.275 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:20.275+0000 7f563b129880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:20.276 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:20.276+0000 7f563b129880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:20.287 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:15:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:15:20.287 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:20.288 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:20.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:20.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025720.077372_98.1", 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:15:20.077372Z", 2023-09-29T22:15:20.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:20.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:15:20.293 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:15:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:20.294 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:15:20Z" 2023-09-29T22:15:20.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:15:20.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025720.077372_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:20.077372Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:15:20Z" } 2023-09-29T22:15:20.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025720.077372_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:20.077372Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:15:20.295 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:20.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.077372_98.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:20.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696025720.088622_101.1 target=1_1696025720.077372_98.1 2023-09-29T22:15:25.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:25.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:25.296 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:25.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:25.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:25.380 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:25.380+0000 7f2ea2da2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:25.381 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:25.381+0000 7f2ea2da2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:25.393 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:15:25.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:25.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:25.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:15:25.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025720.088622_101.1", 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:25.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:15:20.088622Z", 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:15:25.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.399 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:15:25.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:15:25.400 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:15:25.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:15:25.400 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:15:25Z" 2023-09-29T22:15:25.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:15:25.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025720.088622_101.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:20.088622Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:15:25Z" } 2023-09-29T22:15:25.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025720.088622_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:20.088622Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:15:25.401 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:25.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:25.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:15:25.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:15:25.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:25.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:25.402 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:25.454 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:25.455 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:25.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:25.485+0000 7fd7373a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:25.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:25.486+0000 7fd7373a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:25.497 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:15:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:15:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:15:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025720.088622_101.1", 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:15:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:15:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:15:20.088622Z", 2023-09-29T22:15:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:15:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:15:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:15:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:15:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:15:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:15:25Z" 2023-09-29T22:15:25.506 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:15:25.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025720.088622_101.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:20.088622Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:15:25Z" } 2023-09-29T22:15:25.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025720.088622_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:20.088622Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:15:25.507 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:25.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025720.088622_101.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:15:25.508 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:15:25.508 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:15:25.508 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:15:25.508 DEBUG:boto:Method: HEAD 2023-09-29T22:15:25.508 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:15:25.508 DEBUG:boto:Data: 2023-09-29T22:15:25.508 DEBUG:boto:Headers: {} 2023-09-29T22:15:25.509 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:25.509 DEBUG:boto:Port: 8000 2023-09-29T22:15:25.509 DEBUG:boto:Params: {} 2023-09-29T22:15:25.509 DEBUG:boto:Token: None 2023-09-29T22:15:25.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:25 GMT /dmuhsu-16/ 2023-09-29T22:15:25.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GPnW2NGupmmJZlCxc4460Nakltw= 2023-09-29T22:15:25.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GPnW2NGupmmJZlCxc4460Nakltw=', 'Content-Length': '0'} 2023-09-29T22:15:25.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a93e1c9db49ff4d-0065174c7d-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:25.512 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:15:25.512 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:15:25.513 DEBUG:boto:Method: PUT 2023-09-29T22:15:25.513 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:15:25.513 DEBUG:boto:Data: 2023-09-29T22:15:25.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:15:25.513 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:15:25.513 DEBUG:boto:Port: 8000 2023-09-29T22:15:25.514 DEBUG:boto:Params: {} 2023-09-29T22:15:25.514 DEBUG:boto:Token: None 2023-09-29T22:15:25.514 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:15:25 GMT /dmuhsu-16/dummy 2023-09-29T22:15:25.514 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j0eMDE/vWDnS2j70CRYWzad9sdk= 2023-09-29T22:15:25.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j0eMDE/vWDnS2j70CRYWzad9sdk='} 2023-09-29T22:15:25.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-amz-request-id', 'tx000009bb80ba2b12a3163-0065174c7d-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:25.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:25.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:25.525 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:25.579 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:25.579 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:25.611 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:25.611+0000 7f4efcd33880 1 waiting to reach incremental sync.. 2023-09-29T22:15:30.613 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:30.612+0000 7f4efcd33880 1 waiting to reach incremental sync.. 2023-09-29T22:15:35.614 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:35.613+0000 7f4efcd33880 1 bucket sync caught up with source: 2023-09-29T22:15:35.614 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-29T22:15:35.614 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.11.11, , , , , ] 2023-09-29T22:15:35.614 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:15:35.613+0000 7f4efcd33880 0 bucket checkpoint complete 2023-09-29T22:15:35.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:35.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:35.629 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:35.689 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:35.689 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:35.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:35.724+0000 7f608ce15880 1 waiting to reach incremental sync.. 2023-09-29T22:15:40.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:40.725+0000 7f608ce15880 1 bucket sync caught up with source: 2023-09-29T22:15:40.725 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000003.13.10, , , , , ] 2023-09-29T22:15:40.725 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.11.11, , , , , ] 2023-09-29T22:15:40.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:40.725+0000 7f608ce15880 0 bucket checkpoint complete 2023-09-29T22:15:40.738 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:15:40.739 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:15:40.739 DEBUG:boto:Method: HEAD 2023-09-29T22:15:40.739 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:15:40.740 DEBUG:boto:Data: 2023-09-29T22:15:40.740 DEBUG:boto:Headers: {} 2023-09-29T22:15:40.740 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:40.740 DEBUG:boto:Port: 8000 2023-09-29T22:15:40.740 DEBUG:boto:Params: {} 2023-09-29T22:15:40.741 DEBUG:boto:Token: None 2023-09-29T22:15:40.741 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:40 GMT /dmuhsu-17/ 2023-09-29T22:15:40.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XGUsg6AxtH7uw1ggolA3MxkzQYo= 2023-09-29T22:15:40.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XGUsg6AxtH7uw1ggolA3MxkzQYo=', 'Content-Length': '0'} 2023-09-29T22:15:40.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e85a7cc378294c2-0065174c8c-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:40.745 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:15:40.746 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:15:40.746 DEBUG:boto:Method: PUT 2023-09-29T22:15:40.746 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:15:40.746 DEBUG:boto:Data: 2023-09-29T22:15:40.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:15:40.747 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:15:40.747 DEBUG:boto:Port: 8000 2023-09-29T22:15:40.747 DEBUG:boto:Params: {} 2023-09-29T22:15:40.748 DEBUG:boto:Token: None 2023-09-29T22:15:40.748 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:15:40 GMT /dmuhsu-17/dummy 2023-09-29T22:15:40.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mTNdWxqtLyVvXCctwEuzWBe9zHE= 2023-09-29T22:15:40.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mTNdWxqtLyVvXCctwEuzWBe9zHE='} 2023-09-29T22:15:40.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-amz-request-id', 'tx000008a502c6b01e9f158-0065174c8c-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:40.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:40.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:40.762 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:40.816 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:40.816 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:40.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:40.853+0000 7f5eb640a880 1 waiting to reach incremental sync.. 2023-09-29T22:15:45.853 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:45.853+0000 7f5eb640a880 1 waiting to reach incremental sync.. 2023-09-29T22:15:50.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:50.853+0000 7f5eb640a880 1 waiting to reach incremental sync.. 2023-09-29T22:15:55.854 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:15:55.854+0000 7f5eb640a880 1 waiting to reach incremental sync.. 2023-09-29T22:16:00.856 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:00.855+0000 7f5eb640a880 1 bucket sync caught up with source: 2023-09-29T22:16:00.856 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000002.25.11, , , , , ] 2023-09-29T22:16:00.856 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.25.11, , , , , ] 2023-09-29T22:16:00.857 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:00.855+0000 7f5eb640a880 0 bucket checkpoint complete 2023-09-29T22:16:00.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:00.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:00.872 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:16:00.931 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:16:00.931 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:16:00.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:00.968+0000 7fc708826880 1 bucket sync caught up with source: 2023-09-29T22:16:00.969 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000002.25.11, , , , , ] 2023-09-29T22:16:00.969 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.25.11, , , , , ] 2023-09-29T22:16:00.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:00.968+0000 7fc708826880 0 bucket checkpoint complete 2023-09-29T22:16:00.981 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:16:00.981 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:16:00.981 DEBUG:boto:Method: HEAD 2023-09-29T22:16:00.982 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:16:00.982 DEBUG:boto:Data: 2023-09-29T22:16:00.982 DEBUG:boto:Headers: {} 2023-09-29T22:16:00.982 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:00.982 DEBUG:boto:Port: 8001 2023-09-29T22:16:00.982 DEBUG:boto:Params: {} 2023-09-29T22:16:00.982 DEBUG:boto:Token: None 2023-09-29T22:16:00.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:00 GMT /dmuhsu-18/ 2023-09-29T22:16:00.983 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPDUIFgdmmm6aIN4LF3bvGLb4Yk= 2023-09-29T22:16:00.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPDUIFgdmmm6aIN4LF3bvGLb4Yk=', 'Content-Length': '0'} 2023-09-29T22:16:00.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d337f0d4ae80899-0065174ca0-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:00.985 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:16:00.985 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:16:00.986 DEBUG:boto:Method: PUT 2023-09-29T22:16:00.986 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:16:00.986 DEBUG:boto:Data: 2023-09-29T22:16:00.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:16:00.986 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:00.986 DEBUG:boto:Port: 8001 2023-09-29T22:16:00.986 DEBUG:boto:Params: {} 2023-09-29T22:16:00.986 DEBUG:boto:Token: None 2023-09-29T22:16:00.986 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:16:00 GMT /dmuhsu-18/dummy 2023-09-29T22:16:00.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uJgfuN6PWgyQQQc/ro2q4zNwp3k= 2023-09-29T22:16:00.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uJgfuN6PWgyQQQc/ro2q4zNwp3k='} 2023-09-29T22:16:00.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-amz-request-id', 'tx00000efd3609c7d8dcadf-0065174ca0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:00.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:00.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:00.997 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:16:01.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:16:01.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:16:01.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:01.079+0000 7f1429d29880 1 waiting to reach incremental sync.. 2023-09-29T22:16:06.080 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:06.081+0000 7f1429d29880 1 waiting to reach incremental sync.. 2023-09-29T22:16:11.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:11.081+0000 7f1429d29880 1 waiting to reach incremental sync.. 2023-09-29T22:16:16.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:16.083+0000 7f1429d29880 1 bucket sync caught up with source: 2023-09-29T22:16:16.083 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000002.33.11, , , , , ] 2023-09-29T22:16:16.083 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.33.11, , , , , ] 2023-09-29T22:16:16.083 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:16.083+0000 7f1429d29880 0 bucket checkpoint complete 2023-09-29T22:16:16.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:16.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:16.100 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:16:16.159 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:16:16.160 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:16:16.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:16.202+0000 7f8290aba880 1 bucket sync caught up with source: 2023-09-29T22:16:16.202 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.33.11, , , , , ] 2023-09-29T22:16:16.202 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.33.11, , , , , ] 2023-09-29T22:16:16.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:16.202+0000 7f8290aba880 0 bucket checkpoint complete 2023-09-29T22:16:16.217 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:16:16.217 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:16:16.217 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.217 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:16:16.217 DEBUG:boto:Data: 2023-09-29T22:16:16.218 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.218 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.218 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.218 DEBUG:boto:Params: {} 2023-09-29T22:16:16.218 DEBUG:boto:Token: None 2023-09-29T22:16:16.218 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/ 2023-09-29T22:16:16.218 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg= 2023-09-29T22:16:16.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg=', 'Content-Length': '0'} 2023-09-29T22:16:16.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a18463636f3f4bc-0065174cb0-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.221 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.222 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.222 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.222 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.222 DEBUG:boto:Data: 2023-09-29T22:16:16.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.222 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.222 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.222 DEBUG:boto:Params: {} 2023-09-29T22:16:16.222 DEBUG:boto:Token: None 2023-09-29T22:16:16.223 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kwqKl0fdKOCRLqhE1mKimOBuhfw= 2023-09-29T22:16:16.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kwqKl0fdKOCRLqhE1mKimOBuhfw='} 2023-09-29T22:16:16.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-amz-request-id', 'tx0000050c0a1f56a90cae4-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.233 DEBUG:rgw_multi.tests:version1 id=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:16:16.234 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.234 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.234 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.234 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.234 DEBUG:boto:Data: 2023-09-29T22:16:16.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.234 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.234 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.235 DEBUG:boto:Params: {} 2023-09-29T22:16:16.235 DEBUG:boto:Token: None 2023-09-29T22:16:16.235 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.235 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mtvWsApAvEXCo2d7qzMfNFIWDOU= 2023-09-29T22:16:16.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mtvWsApAvEXCo2d7qzMfNFIWDOU='} 2023-09-29T22:16:16.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii'), ('x-amz-request-id', 'tx00000bdd1f6314faf4eb4-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.244 DEBUG:rgw_multi.tests:version2 id=Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii 2023-09-29T22:16:16.244 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.244 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.245 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii 2023-09-29T22:16:16.245 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii 2023-09-29T22:16:16.245 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.245 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii 2023-09-29T22:16:16.245 DEBUG:boto:Data: 2023-09-29T22:16:16.245 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.245 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.245 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.245 DEBUG:boto:Params: {} 2023-09-29T22:16:16.246 DEBUG:boto:Token: None 2023-09-29T22:16:16.246 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone1?versionId=Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii 2023-09-29T22:16:16.246 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MiRiNhuQNZmB8/kYyDqy5y8q2D0= 2023-09-29T22:16:16.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MiRiNhuQNZmB8/kYyDqy5y8q2D0=', 'Content-Length': '0'} 2023-09-29T22:16:16.255 DEBUG:boto:Response headers: [('x-amz-version-id', 'Qu-bzpxBfq.niBjYBxtSa0sMhfZc-Ii'), ('x-amz-request-id', 'tx00000fa9dc9262f8eabd5-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.256 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.256 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.256 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.256 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.256 DEBUG:boto:Data: 2023-09-29T22:16:16.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.256 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.256 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.257 DEBUG:boto:Params: {} 2023-09-29T22:16:16.257 DEBUG:boto:Token: None 2023-09-29T22:16:16.257 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.257 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3lVvKelVNAKn44GK+q2Y4VlYtzs= 2023-09-29T22:16:16.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3lVvKelVNAKn44GK+q2Y4VlYtzs='} 2023-09-29T22:16:16.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym'), ('x-amz-request-id', 'tx0000046c2d8b6293915c9-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.303 DEBUG:rgw_multi.tests:version3 id=w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym 2023-09-29T22:16:16.303 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.303 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:16:16.303 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym 2023-09-29T22:16:16.303 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym 2023-09-29T22:16:16.303 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.303 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym 2023-09-29T22:16:16.304 DEBUG:boto:Data: 2023-09-29T22:16:16.304 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.304 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.304 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.304 DEBUG:boto:Params: {} 2023-09-29T22:16:16.304 DEBUG:boto:Token: None 2023-09-29T22:16:16.304 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone1?versionId=w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym 2023-09-29T22:16:16.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JRgzWav/kNC71QbQd+bKQaWelME= 2023-09-29T22:16:16.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JRgzWav/kNC71QbQd+bKQaWelME=', 'Content-Length': '0'} 2023-09-29T22:16:16.313 DEBUG:boto:Response headers: [('x-amz-version-id', 'w41iQ-MtEIvc3FSm2JnF3Bt2FuKkGym'), ('x-amz-request-id', 'tx0000025b4ef93fce6a6c1-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.313 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:16:16.314 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:16:16.314 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.314 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:16:16.314 DEBUG:boto:Data: 2023-09-29T22:16:16.314 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.314 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.314 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.314 DEBUG:boto:Params: {} 2023-09-29T22:16:16.314 DEBUG:boto:Token: None 2023-09-29T22:16:16.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/ 2023-09-29T22:16:16.315 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg= 2023-09-29T22:16:16.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg=', 'Content-Length': '0'} 2023-09-29T22:16:16.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000248e88bf8132076-0065174cb0-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.317 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.317 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.317 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.318 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.318 DEBUG:boto:Data: 2023-09-29T22:16:16.318 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.318 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.318 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.318 DEBUG:boto:Params: {} 2023-09-29T22:16:16.318 DEBUG:boto:Token: None 2023-09-29T22:16:16.318 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/jZfFMuUGOkuS9BJtgRS2rf5y0M= 2023-09-29T22:16:16.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/jZfFMuUGOkuS9BJtgRS2rf5y0M='} 2023-09-29T22:16:16.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-amz-request-id', 'tx000004c8ad8dbad1c570e-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.332 DEBUG:rgw_multi.tests:version1 id=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:16:16.332 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.332 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.332 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.332 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.332 DEBUG:boto:Data: 2023-09-29T22:16:16.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.333 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.333 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.333 DEBUG:boto:Params: {} 2023-09-29T22:16:16.333 DEBUG:boto:Token: None 2023-09-29T22:16:16.333 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4eUpoQNz2aXmK8K7Xbxq83fnpA8= 2023-09-29T22:16:16.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4eUpoQNz2aXmK8K7Xbxq83fnpA8='} 2023-09-29T22:16:16.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk'), ('x-amz-request-id', 'tx00000ac90e53d87337a79-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.343 DEBUG:rgw_multi.tests:version2 id=GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk 2023-09-29T22:16:16.343 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.343 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.343 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk 2023-09-29T22:16:16.343 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk 2023-09-29T22:16:16.343 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.343 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk 2023-09-29T22:16:16.344 DEBUG:boto:Data: 2023-09-29T22:16:16.344 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.344 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.344 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.344 DEBUG:boto:Params: {} 2023-09-29T22:16:16.344 DEBUG:boto:Token: None 2023-09-29T22:16:16.344 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone2?versionId=GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk 2023-09-29T22:16:16.345 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vy6U7PnQxroJ3+FIMG72QhdslwA= 2023-09-29T22:16:16.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vy6U7PnQxroJ3+FIMG72QhdslwA=', 'Content-Length': '0'} 2023-09-29T22:16:16.353 DEBUG:boto:Response headers: [('x-amz-version-id', 'GF.pZMR0XYIj0GWErx9TuAlo0w-1wLk'), ('x-amz-request-id', 'tx0000000943eee070e3d0a-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.353 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.354 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.354 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.354 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.354 DEBUG:boto:Data: 2023-09-29T22:16:16.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.354 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.354 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.354 DEBUG:boto:Params: {} 2023-09-29T22:16:16.355 DEBUG:boto:Token: None 2023-09-29T22:16:16.355 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.355 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l2DWe0MUmrvAp6fIftRSuzQZ/FQ= 2023-09-29T22:16:16.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l2DWe0MUmrvAp6fIftRSuzQZ/FQ='} 2023-09-29T22:16:16.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '23R7TbeM31qxTiaHKpeoYXvDLwNOEXE'), ('x-amz-request-id', 'tx00000279f1eb2e0b4e308-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.399 DEBUG:rgw_multi.tests:version3 id=23R7TbeM31qxTiaHKpeoYXvDLwNOEXE 2023-09-29T22:16:16.399 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.399 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:16:16.399 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=23R7TbeM31qxTiaHKpeoYXvDLwNOEXE 2023-09-29T22:16:16.399 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=23R7TbeM31qxTiaHKpeoYXvDLwNOEXE 2023-09-29T22:16:16.399 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.399 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=23R7TbeM31qxTiaHKpeoYXvDLwNOEXE 2023-09-29T22:16:16.399 DEBUG:boto:Data: 2023-09-29T22:16:16.400 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.400 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.400 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.400 DEBUG:boto:Params: {} 2023-09-29T22:16:16.400 DEBUG:boto:Token: None 2023-09-29T22:16:16.400 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone2?versionId=23R7TbeM31qxTiaHKpeoYXvDLwNOEXE 2023-09-29T22:16:16.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tNpOu74e/lPwbBqEnJOMLUWAAUE= 2023-09-29T22:16:16.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tNpOu74e/lPwbBqEnJOMLUWAAUE=', 'Content-Length': '0'} 2023-09-29T22:16:16.408 DEBUG:boto:Response headers: [('x-amz-version-id', '23R7TbeM31qxTiaHKpeoYXvDLwNOEXE'), ('x-amz-request-id', 'tx000000335f6b18ad668c0-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.409 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:16:16.409 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:16:16.409 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.409 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:16:16.409 DEBUG:boto:Data: 2023-09-29T22:16:16.409 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.409 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.409 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.409 DEBUG:boto:Params: {} 2023-09-29T22:16:16.410 DEBUG:boto:Token: None 2023-09-29T22:16:16.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/ 2023-09-29T22:16:16.410 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg= 2023-09-29T22:16:16.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7M/syuEmyiV5mFSNDVm+hcWJNLg=', 'Content-Length': '0'} 2023-09-29T22:16:16.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4217bfd94234552-0065174cb0-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.412 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.413 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.413 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.413 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.413 DEBUG:boto:Data: 2023-09-29T22:16:16.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.413 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.413 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.414 DEBUG:boto:Params: {} 2023-09-29T22:16:16.414 DEBUG:boto:Token: None 2023-09-29T22:16:16.414 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.414 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ddw4NzHNUfgBcPrG/WmL9Z1DsbE= 2023-09-29T22:16:16.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ddw4NzHNUfgBcPrG/WmL9Z1DsbE='} 2023-09-29T22:16:16.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-amz-request-id', 'tx00000379f22a202b9a52d-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.423 DEBUG:rgw_multi.tests:version1 id=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:16:16.423 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.423 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.424 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.424 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.424 DEBUG:boto:Data: 2023-09-29T22:16:16.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.424 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.424 DEBUG:boto:Params: {} 2023-09-29T22:16:16.424 DEBUG:boto:Token: None 2023-09-29T22:16:16.424 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zwxBlUOCcH03puW6deFrkPVbhbw= 2023-09-29T22:16:16.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zwxBlUOCcH03puW6deFrkPVbhbw='} 2023-09-29T22:16:16.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rNucuLDXWXODy7eZQeV5ta9bXpm8pTH'), ('x-amz-request-id', 'tx00000b49f8c815b07a865-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.433 DEBUG:rgw_multi.tests:version2 id=rNucuLDXWXODy7eZQeV5ta9bXpm8pTH 2023-09-29T22:16:16.434 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.434 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.434 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=rNucuLDXWXODy7eZQeV5ta9bXpm8pTH 2023-09-29T22:16:16.434 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=rNucuLDXWXODy7eZQeV5ta9bXpm8pTH 2023-09-29T22:16:16.434 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.435 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=rNucuLDXWXODy7eZQeV5ta9bXpm8pTH 2023-09-29T22:16:16.435 DEBUG:boto:Data: 2023-09-29T22:16:16.435 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.435 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.436 DEBUG:boto:Params: {} 2023-09-29T22:16:16.436 DEBUG:boto:Token: None 2023-09-29T22:16:16.436 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone3?versionId=rNucuLDXWXODy7eZQeV5ta9bXpm8pTH 2023-09-29T22:16:16.436 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1LTEcfE6nIRHxB8EtYw1KzzsMls= 2023-09-29T22:16:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1LTEcfE6nIRHxB8EtYw1KzzsMls=', 'Content-Length': '0'} 2023-09-29T22:16:16.445 DEBUG:boto:Response headers: [('x-amz-version-id', 'rNucuLDXWXODy7eZQeV5ta9bXpm8pTH'), ('x-amz-request-id', 'tx00000d2e426353583d4c6-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.445 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.446 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.446 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.446 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.446 DEBUG:boto:Data: 2023-09-29T22:16:16.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.446 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.446 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.446 DEBUG:boto:Params: {} 2023-09-29T22:16:16.446 DEBUG:boto:Token: None 2023-09-29T22:16:16.447 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.447 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6SEqrECvbadKmRYulZQR6bDIVCI= 2023-09-29T22:16:16.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6SEqrECvbadKmRYulZQR6bDIVCI='} 2023-09-29T22:16:16.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI'), ('x-amz-request-id', 'tx00000ff8a9cc1cc4c6e55-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.495 DEBUG:rgw_multi.tests:version3 id=6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI 2023-09-29T22:16:16.495 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.495 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:16:16.495 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI 2023-09-29T22:16:16.495 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI 2023-09-29T22:16:16.495 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.495 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI 2023-09-29T22:16:16.496 DEBUG:boto:Data: 2023-09-29T22:16:16.496 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.496 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.496 DEBUG:boto:Params: {} 2023-09-29T22:16:16.496 DEBUG:boto:Token: None 2023-09-29T22:16:16.496 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-16/obj-test-zone3?versionId=6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI 2023-09-29T22:16:16.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VxVtHeCY6xN28Qq0brSYIAVPnTM= 2023-09-29T22:16:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VxVtHeCY6xN28Qq0brSYIAVPnTM=', 'Content-Length': '0'} 2023-09-29T22:16:16.506 DEBUG:boto:Response headers: [('x-amz-version-id', '6NikIKqAIxLW1wkHJHx9G.yyYO3wvcI'), ('x-amz-request-id', 'tx000006d151277508c352d-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.506 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:16:16.506 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:16:16.506 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.506 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:16:16.507 DEBUG:boto:Data: 2023-09-29T22:16:16.507 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.507 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.507 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.507 DEBUG:boto:Params: {} 2023-09-29T22:16:16.507 DEBUG:boto:Token: None 2023-09-29T22:16:16.507 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/ 2023-09-29T22:16:16.507 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE= 2023-09-29T22:16:16.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE=', 'Content-Length': '0'} 2023-09-29T22:16:16.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf76461c7e143f07-0065174cb0-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.514 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.514 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.514 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.514 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.515 DEBUG:boto:Data: 2023-09-29T22:16:16.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.515 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.515 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.515 DEBUG:boto:Params: {} 2023-09-29T22:16:16.515 DEBUG:boto:Token: None 2023-09-29T22:16:16.515 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bbi225Ip9K17/7+Jzq4FbUXGtGY= 2023-09-29T22:16:16.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bbi225Ip9K17/7+Jzq4FbUXGtGY='} 2023-09-29T22:16:16.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-amz-request-id', 'tx000005aa4a0e971ef6391-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.525 DEBUG:rgw_multi.tests:version1 id=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:16:16.526 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.526 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.526 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.526 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.526 DEBUG:boto:Data: 2023-09-29T22:16:16.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.526 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.526 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.527 DEBUG:boto:Params: {} 2023-09-29T22:16:16.527 DEBUG:boto:Token: None 2023-09-29T22:16:16.527 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.527 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR09B5r3GHJIRhZi4dDuSRyAWW8= 2023-09-29T22:16:16.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR09B5r3GHJIRhZi4dDuSRyAWW8='} 2023-09-29T22:16:16.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u9h.YKBr5XTdrnOtDSDul7byok37dFh'), ('x-amz-request-id', 'tx0000030352e4fa612c639-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.536 DEBUG:rgw_multi.tests:version2 id=u9h.YKBr5XTdrnOtDSDul7byok37dFh 2023-09-29T22:16:16.536 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.536 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.537 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=u9h.YKBr5XTdrnOtDSDul7byok37dFh 2023-09-29T22:16:16.537 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=u9h.YKBr5XTdrnOtDSDul7byok37dFh 2023-09-29T22:16:16.537 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.537 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=u9h.YKBr5XTdrnOtDSDul7byok37dFh 2023-09-29T22:16:16.537 DEBUG:boto:Data: 2023-09-29T22:16:16.537 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.537 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.537 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.537 DEBUG:boto:Params: {} 2023-09-29T22:16:16.537 DEBUG:boto:Token: None 2023-09-29T22:16:16.538 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone1?versionId=u9h.YKBr5XTdrnOtDSDul7byok37dFh 2023-09-29T22:16:16.538 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KVkTE3J9xJEi3LlBsd+sRKq+IPE= 2023-09-29T22:16:16.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KVkTE3J9xJEi3LlBsd+sRKq+IPE=', 'Content-Length': '0'} 2023-09-29T22:16:16.548 DEBUG:boto:Response headers: [('x-amz-version-id', 'u9h.YKBr5XTdrnOtDSDul7byok37dFh'), ('x-amz-request-id', 'tx0000024751ae3f69d1a99-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.548 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.549 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.549 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.549 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.549 DEBUG:boto:Data: 2023-09-29T22:16:16.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.549 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.549 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.549 DEBUG:boto:Params: {} 2023-09-29T22:16:16.550 DEBUG:boto:Token: None 2023-09-29T22:16:16.550 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.550 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P8F6JRQWslglr17p/20BZjLHU4w= 2023-09-29T22:16:16.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P8F6JRQWslglr17p/20BZjLHU4w='} 2023-09-29T22:16:16.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sYD1orVuCyZPSeisYaEOX28SQvWypGu'), ('x-amz-request-id', 'tx00000cd698303de8e4e58-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.595 DEBUG:rgw_multi.tests:version3 id=sYD1orVuCyZPSeisYaEOX28SQvWypGu 2023-09-29T22:16:16.595 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.595 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:16:16.595 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=sYD1orVuCyZPSeisYaEOX28SQvWypGu 2023-09-29T22:16:16.595 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=sYD1orVuCyZPSeisYaEOX28SQvWypGu 2023-09-29T22:16:16.595 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.596 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=sYD1orVuCyZPSeisYaEOX28SQvWypGu 2023-09-29T22:16:16.596 DEBUG:boto:Data: 2023-09-29T22:16:16.596 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.596 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.596 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.596 DEBUG:boto:Params: {} 2023-09-29T22:16:16.596 DEBUG:boto:Token: None 2023-09-29T22:16:16.596 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone1?versionId=sYD1orVuCyZPSeisYaEOX28SQvWypGu 2023-09-29T22:16:16.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RL3UKKSVbemRuge5lHMLCdcGi4E= 2023-09-29T22:16:16.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RL3UKKSVbemRuge5lHMLCdcGi4E=', 'Content-Length': '0'} 2023-09-29T22:16:16.606 DEBUG:boto:Response headers: [('x-amz-version-id', 'sYD1orVuCyZPSeisYaEOX28SQvWypGu'), ('x-amz-request-id', 'tx00000e82e87e6ed023e11-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.606 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:16:16.606 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:16:16.606 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.606 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:16:16.606 DEBUG:boto:Data: 2023-09-29T22:16:16.606 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.607 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.607 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.607 DEBUG:boto:Params: {} 2023-09-29T22:16:16.607 DEBUG:boto:Token: None 2023-09-29T22:16:16.607 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/ 2023-09-29T22:16:16.607 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE= 2023-09-29T22:16:16.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE=', 'Content-Length': '0'} 2023-09-29T22:16:16.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001579f1e41c07ef95-0065174cb0-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.609 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.610 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.610 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.610 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.610 DEBUG:boto:Data: 2023-09-29T22:16:16.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.610 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.610 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.610 DEBUG:boto:Params: {} 2023-09-29T22:16:16.610 DEBUG:boto:Token: None 2023-09-29T22:16:16.611 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.611 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe/DyGOdA8aRrPPOXHz6qxl8fDc= 2023-09-29T22:16:16.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe/DyGOdA8aRrPPOXHz6qxl8fDc='} 2023-09-29T22:16:16.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-amz-request-id', 'tx000008d945f63d9b61bac-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.623 DEBUG:rgw_multi.tests:version1 id=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:16:16.623 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.623 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.623 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.623 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.623 DEBUG:boto:Data: 2023-09-29T22:16:16.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.624 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.624 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.624 DEBUG:boto:Params: {} 2023-09-29T22:16:16.624 DEBUG:boto:Token: None 2023-09-29T22:16:16.624 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:snonSBNse3C3V32MT2wRIAD+Xqg= 2023-09-29T22:16:16.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:snonSBNse3C3V32MT2wRIAD+Xqg='} 2023-09-29T22:16:16.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PNwSbb.tL0gLSNTft4midd9Msg-eoL3'), ('x-amz-request-id', 'tx000004b8a193780d4049f-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.634 DEBUG:rgw_multi.tests:version2 id=PNwSbb.tL0gLSNTft4midd9Msg-eoL3 2023-09-29T22:16:16.634 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.635 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.635 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=PNwSbb.tL0gLSNTft4midd9Msg-eoL3 2023-09-29T22:16:16.635 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=PNwSbb.tL0gLSNTft4midd9Msg-eoL3 2023-09-29T22:16:16.635 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.635 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=PNwSbb.tL0gLSNTft4midd9Msg-eoL3 2023-09-29T22:16:16.635 DEBUG:boto:Data: 2023-09-29T22:16:16.635 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.635 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.635 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.635 DEBUG:boto:Params: {} 2023-09-29T22:16:16.636 DEBUG:boto:Token: None 2023-09-29T22:16:16.636 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone2?versionId=PNwSbb.tL0gLSNTft4midd9Msg-eoL3 2023-09-29T22:16:16.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3ZvoiWlFRF1xQwcAu4VUOTgfVe0= 2023-09-29T22:16:16.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3ZvoiWlFRF1xQwcAu4VUOTgfVe0=', 'Content-Length': '0'} 2023-09-29T22:16:16.644 DEBUG:boto:Response headers: [('x-amz-version-id', 'PNwSbb.tL0gLSNTft4midd9Msg-eoL3'), ('x-amz-request-id', 'tx000004d993a8ad741635f-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.645 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.645 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.645 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.645 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.645 DEBUG:boto:Data: 2023-09-29T22:16:16.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.645 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.646 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.646 DEBUG:boto:Params: {} 2023-09-29T22:16:16.646 DEBUG:boto:Token: None 2023-09-29T22:16:16.646 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.646 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kdkq+T5jiPswGfgvE70/uBvX6k4= 2023-09-29T22:16:16.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kdkq+T5jiPswGfgvE70/uBvX6k4='} 2023-09-29T22:16:16.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf'), ('x-amz-request-id', 'tx0000098dca22f16c756ec-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.691 DEBUG:rgw_multi.tests:version3 id=jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf 2023-09-29T22:16:16.691 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.691 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:16:16.691 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf 2023-09-29T22:16:16.691 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf 2023-09-29T22:16:16.691 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.691 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf 2023-09-29T22:16:16.691 DEBUG:boto:Data: 2023-09-29T22:16:16.692 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.692 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.692 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.692 DEBUG:boto:Params: {} 2023-09-29T22:16:16.692 DEBUG:boto:Token: None 2023-09-29T22:16:16.692 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone2?versionId=jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf 2023-09-29T22:16:16.692 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H2WgNzFJSyPjhm/R1354FLnGsU0= 2023-09-29T22:16:16.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H2WgNzFJSyPjhm/R1354FLnGsU0=', 'Content-Length': '0'} 2023-09-29T22:16:16.702 DEBUG:boto:Response headers: [('x-amz-version-id', 'jdWYR4Kxk9gaQJiIV7VzMkjPUNZxcxf'), ('x-amz-request-id', 'tx000000e1b00836511c0a1-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.702 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:16:16.702 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:16:16.702 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.702 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:16:16.702 DEBUG:boto:Data: 2023-09-29T22:16:16.702 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.703 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.703 DEBUG:boto:Params: {} 2023-09-29T22:16:16.703 DEBUG:boto:Token: None 2023-09-29T22:16:16.703 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/ 2023-09-29T22:16:16.703 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE= 2023-09-29T22:16:16.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Axd/z5HUsBSWbjVxGKeBRtRBPIE=', 'Content-Length': '0'} 2023-09-29T22:16:16.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6512baebcd3f5c9-0065174cb0-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.706 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.706 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.706 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.706 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.706 DEBUG:boto:Data: 2023-09-29T22:16:16.706 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.706 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.706 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.706 DEBUG:boto:Params: {} 2023-09-29T22:16:16.707 DEBUG:boto:Token: None 2023-09-29T22:16:16.707 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.707 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o+ZveBJpAuxgdIgoa6NhwwcrTz8= 2023-09-29T22:16:16.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o+ZveBJpAuxgdIgoa6NhwwcrTz8='} 2023-09-29T22:16:16.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-amz-request-id', 'tx0000056f118f70178fc2c-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.718 DEBUG:rgw_multi.tests:version1 id=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:16:16.718 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.718 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.718 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.718 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.718 DEBUG:boto:Data: 2023-09-29T22:16:16.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.719 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.719 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.719 DEBUG:boto:Params: {} 2023-09-29T22:16:16.719 DEBUG:boto:Token: None 2023-09-29T22:16:16.719 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q3LqyocsnDiDuK9Mm17NmrJjnDQ= 2023-09-29T22:16:16.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q3LqyocsnDiDuK9Mm17NmrJjnDQ='} 2023-09-29T22:16:16.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl'), ('x-amz-request-id', 'tx00000f3f9a6b2c67b9522-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.730 DEBUG:rgw_multi.tests:version2 id=Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl 2023-09-29T22:16:16.731 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.731 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.731 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl 2023-09-29T22:16:16.731 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl 2023-09-29T22:16:16.731 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.731 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl 2023-09-29T22:16:16.731 DEBUG:boto:Data: 2023-09-29T22:16:16.731 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.731 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.732 DEBUG:boto:Params: {} 2023-09-29T22:16:16.732 DEBUG:boto:Token: None 2023-09-29T22:16:16.732 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone3?versionId=Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl 2023-09-29T22:16:16.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OnW+DYB8Yk7ExaKjy+4hL9DDQmQ= 2023-09-29T22:16:16.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OnW+DYB8Yk7ExaKjy+4hL9DDQmQ=', 'Content-Length': '0'} 2023-09-29T22:16:16.740 DEBUG:boto:Response headers: [('x-amz-version-id', 'Qe3ghRb5rDFkBZNoZZLD.tmpdk1VIEl'), ('x-amz-request-id', 'tx00000e3315f17eff23e77-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.740 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.740 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.740 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.740 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.741 DEBUG:boto:Data: 2023-09-29T22:16:16.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.741 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.741 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.741 DEBUG:boto:Params: {} 2023-09-29T22:16:16.741 DEBUG:boto:Token: None 2023-09-29T22:16:16.741 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.741 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iCFF57AWjCBxflR1TVKzUV+/ye0= 2023-09-29T22:16:16.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iCFF57AWjCBxflR1TVKzUV+/ye0='} 2023-09-29T22:16:16.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6pX8wVwsohZDAf0MTcl56.weoMZz9z3'), ('x-amz-request-id', 'tx000001354f71090fd31bf-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.787 DEBUG:rgw_multi.tests:version3 id=6pX8wVwsohZDAf0MTcl56.weoMZz9z3 2023-09-29T22:16:16.787 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.787 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:16:16.787 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=6pX8wVwsohZDAf0MTcl56.weoMZz9z3 2023-09-29T22:16:16.787 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=6pX8wVwsohZDAf0MTcl56.weoMZz9z3 2023-09-29T22:16:16.787 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.787 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=6pX8wVwsohZDAf0MTcl56.weoMZz9z3 2023-09-29T22:16:16.787 DEBUG:boto:Data: 2023-09-29T22:16:16.787 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.788 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.788 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.788 DEBUG:boto:Params: {} 2023-09-29T22:16:16.788 DEBUG:boto:Token: None 2023-09-29T22:16:16.788 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-17/obj-test-zone3?versionId=6pX8wVwsohZDAf0MTcl56.weoMZz9z3 2023-09-29T22:16:16.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oZqhdvrLgs+a63DNBiAsABo3X6U= 2023-09-29T22:16:16.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oZqhdvrLgs+a63DNBiAsABo3X6U=', 'Content-Length': '0'} 2023-09-29T22:16:16.797 DEBUG:boto:Response headers: [('x-amz-version-id', '6pX8wVwsohZDAf0MTcl56.weoMZz9z3'), ('x-amz-request-id', 'tx00000665caf9284a46916-0065174cb0-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.797 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:16:16.797 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:16:16.797 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.797 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:16:16.797 DEBUG:boto:Data: 2023-09-29T22:16:16.797 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.797 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.798 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.798 DEBUG:boto:Params: {} 2023-09-29T22:16:16.798 DEBUG:boto:Token: None 2023-09-29T22:16:16.798 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/ 2023-09-29T22:16:16.798 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8= 2023-09-29T22:16:16.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8=', 'Content-Length': '0'} 2023-09-29T22:16:16.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fa4f1e02d754582-0065174cb0-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.800 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.801 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.801 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.801 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.801 DEBUG:boto:Data: 2023-09-29T22:16:16.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.801 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.801 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.801 DEBUG:boto:Params: {} 2023-09-29T22:16:16.801 DEBUG:boto:Token: None 2023-09-29T22:16:16.802 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FWSwynqRBPh2A4IyltmVm1xyl2Q= 2023-09-29T22:16:16.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FWSwynqRBPh2A4IyltmVm1xyl2Q='} 2023-09-29T22:16:16.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-amz-request-id', 'tx0000012b9cd6dfdd500a7-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.812 DEBUG:rgw_multi.tests:version1 id=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:16:16.812 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.812 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.812 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.813 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.813 DEBUG:boto:Data: 2023-09-29T22:16:16.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.813 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.813 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.813 DEBUG:boto:Params: {} 2023-09-29T22:16:16.813 DEBUG:boto:Token: None 2023-09-29T22:16:16.813 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.813 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NUOuy3FEZDEvVv8fiqGBGUxfQ6E= 2023-09-29T22:16:16.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NUOuy3FEZDEvVv8fiqGBGUxfQ6E='} 2023-09-29T22:16:16.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qG52MmjnIrjLtIKCpibQtbn5HO3HMoY'), ('x-amz-request-id', 'tx000001baea827e089719b-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.824 DEBUG:rgw_multi.tests:version2 id=qG52MmjnIrjLtIKCpibQtbn5HO3HMoY 2023-09-29T22:16:16.824 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.825 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.825 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=qG52MmjnIrjLtIKCpibQtbn5HO3HMoY 2023-09-29T22:16:16.825 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=qG52MmjnIrjLtIKCpibQtbn5HO3HMoY 2023-09-29T22:16:16.825 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.825 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=qG52MmjnIrjLtIKCpibQtbn5HO3HMoY 2023-09-29T22:16:16.825 DEBUG:boto:Data: 2023-09-29T22:16:16.825 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.825 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.825 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.825 DEBUG:boto:Params: {} 2023-09-29T22:16:16.826 DEBUG:boto:Token: None 2023-09-29T22:16:16.826 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone1?versionId=qG52MmjnIrjLtIKCpibQtbn5HO3HMoY 2023-09-29T22:16:16.826 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YHwyZoQm/t28jpSwSZg0oYbF3kQ= 2023-09-29T22:16:16.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YHwyZoQm/t28jpSwSZg0oYbF3kQ=', 'Content-Length': '0'} 2023-09-29T22:16:16.834 DEBUG:boto:Response headers: [('x-amz-version-id', 'qG52MmjnIrjLtIKCpibQtbn5HO3HMoY'), ('x-amz-request-id', 'tx000008f6c385ad62139f6-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.835 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.835 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.835 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.835 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.835 DEBUG:boto:Data: 2023-09-29T22:16:16.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.836 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.836 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.836 DEBUG:boto:Params: {} 2023-09-29T22:16:16.836 DEBUG:boto:Token: None 2023-09-29T22:16:16.836 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dTpNkIQqB0ryVU/sKyJLYf6Tk5U= 2023-09-29T22:16:16.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dTpNkIQqB0ryVU/sKyJLYf6Tk5U='} 2023-09-29T22:16:16.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT'), ('x-amz-request-id', 'tx00000944f7c3c8b8f42a8-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.883 DEBUG:rgw_multi.tests:version3 id=YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT 2023-09-29T22:16:16.883 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.883 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:16:16.883 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT 2023-09-29T22:16:16.883 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT 2023-09-29T22:16:16.883 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.884 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT 2023-09-29T22:16:16.884 DEBUG:boto:Data: 2023-09-29T22:16:16.884 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.884 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:16:16.884 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.884 DEBUG:boto:Params: {} 2023-09-29T22:16:16.884 DEBUG:boto:Token: None 2023-09-29T22:16:16.884 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone1?versionId=YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT 2023-09-29T22:16:16.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f5dZJLOhpJoq8iHyB6YaQoFg1M0= 2023-09-29T22:16:16.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f5dZJLOhpJoq8iHyB6YaQoFg1M0=', 'Content-Length': '0'} 2023-09-29T22:16:16.893 DEBUG:boto:Response headers: [('x-amz-version-id', 'YPqhPejcIi8LfL13GbKZDOZ.rl1FUAT'), ('x-amz-request-id', 'tx00000073ad824b39d711d-0065174cb0-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.893 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:16:16.893 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:16:16.894 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.894 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:16:16.894 DEBUG:boto:Data: 2023-09-29T22:16:16.894 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.894 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.894 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.894 DEBUG:boto:Params: {} 2023-09-29T22:16:16.894 DEBUG:boto:Token: None 2023-09-29T22:16:16.895 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/ 2023-09-29T22:16:16.895 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8= 2023-09-29T22:16:16.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8=', 'Content-Length': '0'} 2023-09-29T22:16:16.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063820f345161af57-0065174cb0-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.897 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.897 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.897 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.897 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.897 DEBUG:boto:Data: 2023-09-29T22:16:16.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.898 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.898 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.898 DEBUG:boto:Params: {} 2023-09-29T22:16:16.898 DEBUG:boto:Token: None 2023-09-29T22:16:16.898 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.898 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MLwDQObQJVp3SnhU2ELbX+k3A/0= 2023-09-29T22:16:16.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MLwDQObQJVp3SnhU2ELbX+k3A/0='} 2023-09-29T22:16:16.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-amz-request-id', 'tx00000de44379fa83268ba-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.912 DEBUG:rgw_multi.tests:version1 id=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:16:16.912 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.912 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.913 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.913 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.913 DEBUG:boto:Data: 2023-09-29T22:16:16.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.913 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.913 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.913 DEBUG:boto:Params: {} 2023-09-29T22:16:16.913 DEBUG:boto:Token: None 2023-09-29T22:16:16.913 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.914 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zwkMNnc7y29xlQ+0k021mxDTOQk= 2023-09-29T22:16:16.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zwkMNnc7y29xlQ+0k021mxDTOQk='} 2023-09-29T22:16:16.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ'), ('x-amz-request-id', 'tx0000032ce12aca32062c5-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.925 DEBUG:rgw_multi.tests:version2 id=uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ 2023-09-29T22:16:16.925 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.925 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.926 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ 2023-09-29T22:16:16.926 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ 2023-09-29T22:16:16.926 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.926 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ 2023-09-29T22:16:16.926 DEBUG:boto:Data: 2023-09-29T22:16:16.926 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.926 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.926 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.926 DEBUG:boto:Params: {} 2023-09-29T22:16:16.926 DEBUG:boto:Token: None 2023-09-29T22:16:16.927 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone2?versionId=uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ 2023-09-29T22:16:16.927 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sSUuYX5uHk69BZ5MAylh7Glf/lE= 2023-09-29T22:16:16.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sSUuYX5uHk69BZ5MAylh7Glf/lE=', 'Content-Length': '0'} 2023-09-29T22:16:16.936 DEBUG:boto:Response headers: [('x-amz-version-id', 'uQR8FTItVZjRwvvqC3gRVcj8LjlprgQ'), ('x-amz-request-id', 'tx000007c8a6082cf64b6db-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.937 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.937 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.937 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.937 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.937 DEBUG:boto:Data: 2023-09-29T22:16:16.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.937 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.937 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.937 DEBUG:boto:Params: {} 2023-09-29T22:16:16.938 DEBUG:boto:Token: None 2023-09-29T22:16:16.938 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.938 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PQKPW+FtHD4j02mvZTSNlEtnUiA= 2023-09-29T22:16:16.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PQKPW+FtHD4j02mvZTSNlEtnUiA='} 2023-09-29T22:16:16.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD'), ('x-amz-request-id', 'tx00000ca4b2acc70a4649a-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.983 DEBUG:rgw_multi.tests:version3 id=vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD 2023-09-29T22:16:16.983 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.983 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:16:16.984 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD 2023-09-29T22:16:16.984 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD 2023-09-29T22:16:16.984 DEBUG:boto:Method: DELETE 2023-09-29T22:16:16.984 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD 2023-09-29T22:16:16.984 DEBUG:boto:Data: 2023-09-29T22:16:16.984 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.985 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:16:16.985 DEBUG:boto:Port: 8000 2023-09-29T22:16:16.985 DEBUG:boto:Params: {} 2023-09-29T22:16:16.985 DEBUG:boto:Token: None 2023-09-29T22:16:16.986 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/obj-test-zone2?versionId=vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD 2023-09-29T22:16:16.986 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k+PFpVBK9prU+Q/L6hsNPA7kgqM= 2023-09-29T22:16:16.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k+PFpVBK9prU+Q/L6hsNPA7kgqM=', 'Content-Length': '0'} 2023-09-29T22:16:16.994 DEBUG:boto:Response headers: [('x-amz-version-id', 'vJUh3rwRh5nZvWqGg5rWSwFXsaO3UaD'), ('x-amz-request-id', 'tx00000c0a97228e6690f50-0065174cb0-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.994 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:16:16.994 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:16:16.994 DEBUG:boto:Method: HEAD 2023-09-29T22:16:16.994 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:16:16.995 DEBUG:boto:Data: 2023-09-29T22:16:16.995 DEBUG:boto:Headers: {} 2023-09-29T22:16:16.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.995 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.995 DEBUG:boto:Params: {} 2023-09-29T22:16:16.995 DEBUG:boto:Token: None 2023-09-29T22:16:16.995 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:16:16 GMT /dmuhsu-18/ 2023-09-29T22:16:16.995 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8= 2023-09-29T22:16:16.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:16 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9C0S2RjhSq3sWh3MUogEZL2F/E8=', 'Content-Length': '0'} 2023-09-29T22:16:16.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003a5a08ecf099a14-0065174cb0-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:16.998 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:16.998 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:16.999 DEBUG:boto:Method: PUT 2023-09-29T22:16:16.999 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:16.999 DEBUG:boto:Data: 2023-09-29T22:16:16.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:16.999 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:16.999 DEBUG:boto:Port: 8001 2023-09-29T22:16:16.999 DEBUG:boto:Params: {} 2023-09-29T22:16:16.999 DEBUG:boto:Token: None 2023-09-29T22:16:17.000 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:16:17 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kFvDFbefq4/NF8HC2uJOJej1SN0= 2023-09-29T22:16:17.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kFvDFbefq4/NF8HC2uJOJej1SN0='} 2023-09-29T22:16:17.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-amz-request-id', 'tx000003acdf701e4f02c31-0065174cb1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:17.010 DEBUG:rgw_multi.tests:version1 id=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:16:17.010 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.011 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.011 DEBUG:boto:Method: PUT 2023-09-29T22:16:17.011 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.011 DEBUG:boto:Data: 2023-09-29T22:16:17.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:17.011 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:17.011 DEBUG:boto:Port: 8001 2023-09-29T22:16:17.011 DEBUG:boto:Params: {} 2023-09-29T22:16:17.012 DEBUG:boto:Token: None 2023-09-29T22:16:17.012 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 22:16:17 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.012 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9uq7QKL3OaCiVuwXpxyPBUmqKs= 2023-09-29T22:16:17.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9uq7QKL3OaCiVuwXpxyPBUmqKs='} 2023-09-29T22:16:17.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg'), ('x-amz-request-id', 'tx00000e264ca31849d812a-0065174cb1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:17.020 DEBUG:rgw_multi.tests:version2 id=Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg 2023-09-29T22:16:17.020 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.021 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.021 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg 2023-09-29T22:16:17.021 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg 2023-09-29T22:16:17.021 DEBUG:boto:Method: DELETE 2023-09-29T22:16:17.021 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg 2023-09-29T22:16:17.021 DEBUG:boto:Data: 2023-09-29T22:16:17.022 DEBUG:boto:Headers: {} 2023-09-29T22:16:17.022 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:17.022 DEBUG:boto:Port: 8001 2023-09-29T22:16:17.022 DEBUG:boto:Params: {} 2023-09-29T22:16:17.022 DEBUG:boto:Token: None 2023-09-29T22:16:17.023 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:17 GMT /dmuhsu-18/obj-test-zone3?versionId=Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg 2023-09-29T22:16:17.023 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ICYeR3W0xJbpLWniMLxgBGngAPU= 2023-09-29T22:16:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ICYeR3W0xJbpLWniMLxgBGngAPU=', 'Content-Length': '0'} 2023-09-29T22:16:17.031 DEBUG:boto:Response headers: [('x-amz-version-id', 'Kzo10obqoEDXRhLhEtSLEnXsRGiQAeg'), ('x-amz-request-id', 'tx000006f2eed9ba728fbbc-0065174cb1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:17.032 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.032 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.032 DEBUG:boto:Method: PUT 2023-09-29T22:16:17.032 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.032 DEBUG:boto:Data: 2023-09-29T22:16:17.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:16:17.032 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:17.033 DEBUG:boto:Port: 8001 2023-09-29T22:16:17.033 DEBUG:boto:Params: {} 2023-09-29T22:16:17.033 DEBUG:boto:Token: None 2023-09-29T22:16:17.033 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 22:16:17 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.033 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:23+JmsBg+bEteV1FgwV5Amf9Mu8= 2023-09-29T22:16:17.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:16:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:23+JmsBg+bEteV1FgwV5Amf9Mu8='} 2023-09-29T22:16:17.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5'), ('x-amz-request-id', 'tx00000df3c1e5acaecccc6-0065174cb1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:17.075 DEBUG:rgw_multi.tests:version3 id=Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5 2023-09-29T22:16:17.075 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.075 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:16:17.075 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5 2023-09-29T22:16:17.075 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5 2023-09-29T22:16:17.075 DEBUG:boto:Method: DELETE 2023-09-29T22:16:17.075 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5 2023-09-29T22:16:17.076 DEBUG:boto:Data: 2023-09-29T22:16:17.076 DEBUG:boto:Headers: {} 2023-09-29T22:16:17.076 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:16:17.076 DEBUG:boto:Port: 8001 2023-09-29T22:16:17.076 DEBUG:boto:Params: {} 2023-09-29T22:16:17.076 DEBUG:boto:Token: None 2023-09-29T22:16:17.076 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:16:17 GMT /dmuhsu-18/obj-test-zone3?versionId=Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5 2023-09-29T22:16:17.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3BrwXuszk+fg/Bm3gMt+yGFotsg= 2023-09-29T22:16:17.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:16:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3BrwXuszk+fg/Bm3gMt+yGFotsg=', 'Content-Length': '0'} 2023-09-29T22:16:17.084 DEBUG:boto:Response headers: [('x-amz-version-id', 'Cdz6lPC7X2sSB8mVH-hqBj4KY6sjys5'), ('x-amz-request-id', 'tx00000f1a8d7dd4148888b-0065174cb1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:16:17.085 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-16 2023-09-29T22:16:17.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:17.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:17.085 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:16:17.138 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:16:17.138 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:16:17.174 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:17.174+0000 7f8cac214880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:17.175 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.15.10, , , , , ] 2023-09-29T22:16:17.175 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:22.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:22.176+0000 7f8cac214880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:22.176 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.15.10, , , , , ] 2023-09-29T22:16:22.176 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:27.177 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:27.177+0000 7f8cac214880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:27.177 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.15.10, , , , , ] 2023-09-29T22:16:27.177 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:32.178 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:32.178+0000 7f8cac214880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:32.244 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.15.10, , , , , ] 2023-09-29T22:16:32.244 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:37.179 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:37.179+0000 7f8cac214880 1 bucket sync caught up with source: 2023-09-29T22:16:37.287 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:37.288 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:37.288 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:16:37.179+0000 7f8cac214880 0 bucket checkpoint complete 2023-09-29T22:16:37.289 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-16 2023-09-29T22:16:37.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:37.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:16:37.290 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:16:37.348 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:16:37.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:16:37.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:37.384+0000 7ff07ae15880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:37.436 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:37.436 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:42.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:42.386+0000 7ff07ae15880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:42.386 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:42.386 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:47.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:47.386+0000 7ff07ae15880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:47.387 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:47.387 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:52.388 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:52.387+0000 7ff07ae15880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:52.388 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:52.388 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:57.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:16:57.389+0000 7ff07ae15880 1 waiting for incremental sync to catch up: 2023-09-29T22:16:57.390 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000002.12.11, , 00000000005.17.10, , , , , ] 2023-09-29T22:16:57.390 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:17:02.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:02.390+0000 7ff07ae15880 1 bucket sync caught up with source: 2023-09-29T22:17:02.391 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:17:02.391 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.26.6, , 00000000003.14.10, , 00000000005.17.10, , , , , ] 2023-09-29T22:17:02.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:02.390+0000 7ff07ae15880 0 bucket checkpoint complete 2023-09-29T22:17:02.406 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-16 2023-09-29T22:17:02.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:02.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:02.406 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:02.463 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:02.464 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:02.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:02.499+0000 7f8b4994b880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:02.500 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000004.43.10, , , , , ] 2023-09-29T22:17:02.501 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000006.47.11, , , , , ] 2023-09-29T22:17:07.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:07.502+0000 7f8b4994b880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:07.502 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000004.43.10, , , , , ] 2023-09-29T22:17:07.502 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000006.47.11, , , , , ] 2023-09-29T22:17:12.503 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:12.503+0000 7f8b4994b880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:12.503 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000004.43.10, , , , , ] 2023-09-29T22:17:12.503 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000006.47.11, , , , , ] 2023-09-29T22:17:16.435 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:17:16.435+0000 7f303f796640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:17:17.504 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:17.505+0000 7f8b4994b880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:17.505 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000004.43.10, , , , , ] 2023-09-29T22:17:17.505 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000006.47.11, , , , , ] 2023-09-29T22:17:17.903 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:17:17.904+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:17:22.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.506+0000 7f8b4994b880 1 bucket sync caught up with source: 2023-09-29T22:17:22.506 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:17:22.506 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000006.47.11, , , , , ] 2023-09-29T22:17:22.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.506+0000 7f8b4994b880 0 bucket checkpoint complete 2023-09-29T22:17:22.521 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-16 2023-09-29T22:17:22.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.522 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:22.580 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:22.580 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:22.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.615+0000 7f7129c19880 1 bucket sync caught up with source: 2023-09-29T22:17:22.615 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000010.55.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:17:22.615 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000010.55.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:17:22.615 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.615+0000 7f7129c19880 0 bucket checkpoint complete 2023-09-29T22:17:22.627 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-16 2023-09-29T22:17:22.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.628 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:22.679 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:22.680 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.712+0000 7f394db20880 1 bucket sync caught up with source: 2023-09-29T22:17:22.712 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:17:22.712 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:17:22.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:22.712+0000 7f394db20880 0 bucket checkpoint complete 2023-09-29T22:17:22.725 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-16 2023-09-29T22:17:22.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:22.725 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:22.784 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:22.784 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:22.820 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:22.820+0000 7fb217f30880 1 bucket sync caught up with source: 2023-09-29T22:17:22.820 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:17:22.821 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:17:22.821 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:22.820+0000 7fb217f30880 0 bucket checkpoint complete 2023-09-29T22:17:22.833 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone2, test-zone1} 2023-09-29T22:17:22.833 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.833 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.834 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.834 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:22.834 DEBUG:boto:Data: 2023-09-29T22:17:22.834 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.834 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.834 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.834 DEBUG:boto:Params: {} 2023-09-29T22:17:22.835 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:17:22.835 DEBUG:boto:Token: None 2023-09-29T22:17:22.835 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/ 2023-09-29T22:17:22.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ayJfhuDbPVgtORmpuyoND3ivplE= 2023-09-29T22:17:22.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ayJfhuDbPVgtORmpuyoND3ivplE=', 'Content-Length': '0'} 2023-09-29T22:17:22.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082f8d9da336bd8e6-0065174cf2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.839 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.839 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.839 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.840 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:22.840 DEBUG:boto:Data: 2023-09-29T22:17:22.840 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.840 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.840 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.840 DEBUG:boto:Params: {} 2023-09-29T22:17:22.840 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:17:22.841 DEBUG:boto:Token: None 2023-09-29T22:17:22.841 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/ 2023-09-29T22:17:22.841 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ayJfhuDbPVgtORmpuyoND3ivplE= 2023-09-29T22:17:22.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ayJfhuDbPVgtORmpuyoND3ivplE=', 'Content-Length': '0'} 2023-09-29T22:17:22.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca94ccea33e259ec-0065174cf2-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.845 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:22.845 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.845 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.845 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.845 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.845 DEBUG:boto:Method: GET 2023-09-29T22:17:22.845 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.845 DEBUG:boto:Data: 2023-09-29T22:17:22.845 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.846 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.846 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.846 DEBUG:boto:Params: {} 2023-09-29T22:17:22.846 DEBUG:boto:Token: None 2023-09-29T22:17:22.846 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/?versions 2023-09-29T22:17:22.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo= 2023-09-29T22:17:22.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo=', 'Content-Length': '0'} 2023-09-29T22:17:22.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e77dc67746ee02fb-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.849 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.849 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:22.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:22.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:22.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:22.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:22.850 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.850 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.850 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.850 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.851 DEBUG:boto:Method: GET 2023-09-29T22:17:22.851 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.851 DEBUG:boto:Data: 2023-09-29T22:17:22.851 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.851 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.851 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.851 DEBUG:boto:Params: {} 2023-09-29T22:17:22.851 DEBUG:boto:Token: None 2023-09-29T22:17:22.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/?versions 2023-09-29T22:17:22.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo= 2023-09-29T22:17:22.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo=', 'Content-Length': '0'} 2023-09-29T22:17:22.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb0f437161ef605f-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.854 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.854 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:22.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:22.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:22.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:22.855 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.855 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.855 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.855 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.855 DEBUG:boto:Method: GET 2023-09-29T22:17:22.855 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.855 DEBUG:boto:Data: 2023-09-29T22:17:22.855 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.856 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.856 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.856 DEBUG:boto:Params: {} 2023-09-29T22:17:22.856 DEBUG:boto:Token: None 2023-09-29T22:17:22.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/?versions 2023-09-29T22:17:22.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo= 2023-09-29T22:17:22.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo=', 'Content-Length': '0'} 2023-09-29T22:17:22.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d79dee9424d006a-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.858 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.860 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.860 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.860 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.861 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.861 DEBUG:boto:Method: GET 2023-09-29T22:17:22.861 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:22.861 DEBUG:boto:Data: 2023-09-29T22:17:22.861 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.861 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.861 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.861 DEBUG:boto:Params: {} 2023-09-29T22:17:22.861 DEBUG:boto:Token: None 2023-09-29T22:17:22.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/?versions 2023-09-29T22:17:22.862 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo= 2023-09-29T22:17:22.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:topsJ6kn0RU5/2eOSqTqqsiIduo=', 'Content-Length': '0'} 2023-09-29T22:17:22.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a84b4f3d2f6f072-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.864 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:22.865 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.865 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.865 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.865 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.865 DEBUG:boto:Method: GET 2023-09-29T22:17:22.865 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.865 DEBUG:boto:Data: 2023-09-29T22:17:22.866 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.866 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.866 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.866 DEBUG:boto:Params: {} 2023-09-29T22:17:22.866 DEBUG:boto:Token: None 2023-09-29T22:17:22.866 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.866 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca8f1f014a3fd98c-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.868 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.868 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.868 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.868 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.869 DEBUG:boto:Method: GET 2023-09-29T22:17:22.869 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.869 DEBUG:boto:Data: 2023-09-29T22:17:22.869 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.869 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.869 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.869 DEBUG:boto:Params: {} 2023-09-29T22:17:22.870 DEBUG:boto:Token: None 2023-09-29T22:17:22.870 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ded0a3d1ac2745be-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.872 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.872 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.872 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.872 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.872 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.872 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.872 DEBUG:boto:Data: 2023-09-29T22:17:22.872 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.872 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.873 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.873 DEBUG:boto:Params: {} 2023-09-29T22:17:22.873 DEBUG:boto:Token: None 2023-09-29T22:17:22.873 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.873 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WvQjhll+nN/3wLXAdoZ4r+Q+RTY= 2023-09-29T22:17:22.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WvQjhll+nN/3wLXAdoZ4r+Q+RTY=', 'Content-Length': '0'} 2023-09-29T22:17:22.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006696530113ef32c2-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.875 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.875 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.875 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.876 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.876 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.876 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.876 DEBUG:boto:Data: 2023-09-29T22:17:22.876 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.876 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.876 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.876 DEBUG:boto:Params: {} 2023-09-29T22:17:22.876 DEBUG:boto:Token: None 2023-09-29T22:17:22.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.877 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WvQjhll+nN/3wLXAdoZ4r+Q+RTY= 2023-09-29T22:17:22.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WvQjhll+nN/3wLXAdoZ4r+Q+RTY=', 'Content-Length': '0'} 2023-09-29T22:17:22.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abc5f28f725a495d-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:22.879 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.879 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.879 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.879 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.879 DEBUG:boto:Method: GET 2023-09-29T22:17:22.879 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.879 DEBUG:boto:Data: 2023-09-29T22:17:22.879 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.879 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.879 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.880 DEBUG:boto:Params: {} 2023-09-29T22:17:22.880 DEBUG:boto:Token: None 2023-09-29T22:17:22.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fba9fa00ba25ca0-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.882 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.882 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.882 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.882 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.882 DEBUG:boto:Method: GET 2023-09-29T22:17:22.883 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.883 DEBUG:boto:Data: 2023-09-29T22:17:22.883 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.883 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.883 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.883 DEBUG:boto:Params: {} 2023-09-29T22:17:22.883 DEBUG:boto:Token: None 2023-09-29T22:17:22.883 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.883 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b28e553c72c55d54-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.886 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.886 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.886 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.886 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.886 DEBUG:boto:Method: GET 2023-09-29T22:17:22.886 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.886 DEBUG:boto:Data: 2023-09-29T22:17:22.887 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.887 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.887 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.887 DEBUG:boto:Params: {} 2023-09-29T22:17:22.887 DEBUG:boto:Token: None 2023-09-29T22:17:22.887 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/ 2023-09-29T22:17:22.887 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0= 2023-09-29T22:17:22.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0=', 'Content-Length': '0'} 2023-09-29T22:17:22.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e622b61b493c32ff-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.930 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.931 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.931 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.931 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.931 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.931 DEBUG:boto:Method: GET 2023-09-29T22:17:22.932 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.932 DEBUG:boto:Data: 2023-09-29T22:17:22.932 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.932 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.932 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.932 DEBUG:boto:Params: {} 2023-09-29T22:17:22.932 DEBUG:boto:Token: None 2023-09-29T22:17:22.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/ 2023-09-29T22:17:22.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0= 2023-09-29T22:17:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0=', 'Content-Length': '0'} 2023-09-29T22:17:22.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df3cc2c61f3f565a-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.935 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.935 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.936 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.936 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.936 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:22.936 DEBUG:boto:Data: 2023-09-29T22:17:22.936 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.936 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.936 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.936 DEBUG:boto:Params: {} 2023-09-29T22:17:22.937 DEBUG:boto:Token: None 2023-09-29T22:17:22.937 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy 2023-09-29T22:17:22.937 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x8gDXKCTGci46Cw8CakCcAQZ6NY= 2023-09-29T22:17:22.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x8gDXKCTGci46Cw8CakCcAQZ6NY=', 'Content-Length': '0'} 2023-09-29T22:17:22.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d0a2f689b34ceca-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.939 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.939 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.939 DEBUG:boto:Method: HEAD 2023-09-29T22:17:22.940 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:22.940 DEBUG:boto:Data: 2023-09-29T22:17:22.940 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.940 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.940 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.940 DEBUG:boto:Params: {} 2023-09-29T22:17:22.940 DEBUG:boto:Token: None 2023-09-29T22:17:22.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy 2023-09-29T22:17:22.941 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x8gDXKCTGci46Cw8CakCcAQZ6NY= 2023-09-29T22:17:22.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x8gDXKCTGci46Cw8CakCcAQZ6NY=', 'Content-Length': '0'} 2023-09-29T22:17:22.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009919220e9db3edeb-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:22.943 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.943 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.944 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.944 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.944 DEBUG:boto:Method: GET 2023-09-29T22:17:22.944 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.944 DEBUG:boto:Data: 2023-09-29T22:17:22.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.944 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.944 DEBUG:boto:Params: {} 2023-09-29T22:17:22.945 DEBUG:boto:Token: None 2023-09-29T22:17:22.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000160a81383f299c17-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.947 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:22.947 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:22.947 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.947 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.948 DEBUG:boto:Method: GET 2023-09-29T22:17:22.948 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.948 DEBUG:boto:Data: 2023-09-29T22:17:22.948 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:22.948 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.948 DEBUG:boto:Params: {} 2023-09-29T22:17:22.948 DEBUG:boto:Token: None 2023-09-29T22:17:22.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:22.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk= 2023-09-29T22:17:22.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G+shnpNYliSLFP7auTOl5Y2nKqk=', 'Content-Length': '0'} 2023-09-29T22:17:22.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ef81e472b3921a6-0065174cf2-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.951 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.951 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.951 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.951 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.951 DEBUG:boto:Method: GET 2023-09-29T22:17:22.951 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.951 DEBUG:boto:Data: 2023-09-29T22:17:22.951 DEBUG:boto:Headers: {} 2023-09-29T22:17:22.952 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:22.952 DEBUG:boto:Port: 8000 2023-09-29T22:17:22.952 DEBUG:boto:Params: {} 2023-09-29T22:17:22.952 DEBUG:boto:Token: None 2023-09-29T22:17:22.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:22 GMT /dmuhsu-16/ 2023-09-29T22:17:22.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0= 2023-09-29T22:17:22.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:16z9A85GaPchLmWMnq27bDBHCJ0=', 'Content-Length': '0'} 2023-09-29T22:17:22.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008504ca1e66d21aa7-0065174cf2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:22.998 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:22.999 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:22.999 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:22.999 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.999 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:22.999 DEBUG:boto:Method: GET 2023-09-29T22:17:22.999 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.000 DEBUG:boto:Data: 2023-09-29T22:17:23.000 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.000 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.000 DEBUG:boto:Params: {} 2023-09-29T22:17:23.000 DEBUG:boto:Token: None 2023-09-29T22:17:23.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.001 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008474a854fbde7afa-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.003 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.005 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.005 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.005 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.005 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.005 DEBUG:boto:Method: GET 2023-09-29T22:17:23.006 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.006 DEBUG:boto:Data: 2023-09-29T22:17:23.006 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.006 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.006 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.006 DEBUG:boto:Params: {} 2023-09-29T22:17:23.006 DEBUG:boto:Token: None 2023-09-29T22:17:23.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005628c96146d879a8-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.050 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.051 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.051 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.051 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.051 DEBUG:boto:Method: GET 2023-09-29T22:17:23.051 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.051 DEBUG:boto:Data: 2023-09-29T22:17:23.051 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.051 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.051 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.051 DEBUG:boto:Params: {} 2023-09-29T22:17:23.052 DEBUG:boto:Token: None 2023-09-29T22:17:23.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a848f70bfb854746-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.054 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.054 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.054 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.054 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.054 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.054 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.054 DEBUG:boto:Data: 2023-09-29T22:17:23.055 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.055 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.055 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.055 DEBUG:boto:Params: {} 2023-09-29T22:17:23.055 DEBUG:boto:Token: None 2023-09-29T22:17:23.055 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0= 2023-09-29T22:17:23.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0=', 'Content-Length': '0'} 2023-09-29T22:17:23.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbbcd30be497f9d2-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.057 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.058 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.058 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.058 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.058 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.058 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.058 DEBUG:boto:Data: 2023-09-29T22:17:23.059 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.059 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.059 DEBUG:boto:Params: {} 2023-09-29T22:17:23.059 DEBUG:boto:Token: None 2023-09-29T22:17:23.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.059 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0= 2023-09-29T22:17:23.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0=', 'Content-Length': '0'} 2023-09-29T22:17:23.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ad7bf1b495c496-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.061 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.061 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.061 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.061 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.061 DEBUG:boto:Method: GET 2023-09-29T22:17:23.062 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.062 DEBUG:boto:Data: 2023-09-29T22:17:23.062 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.062 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.062 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.062 DEBUG:boto:Params: {} 2023-09-29T22:17:23.062 DEBUG:boto:Token: None 2023-09-29T22:17:23.062 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.062 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b7f079bd39e5b21-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.106 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.107 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.107 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.107 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.107 DEBUG:boto:Method: GET 2023-09-29T22:17:23.107 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.107 DEBUG:boto:Data: 2023-09-29T22:17:23.107 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.108 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.108 DEBUG:boto:Params: {} 2023-09-29T22:17:23.108 DEBUG:boto:Token: None 2023-09-29T22:17:23.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b8fe4fc79c5b3ff-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.110 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.111 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.111 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.111 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.111 DEBUG:boto:Method: GET 2023-09-29T22:17:23.111 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.111 DEBUG:boto:Data: 2023-09-29T22:17:23.111 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.111 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.112 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.112 DEBUG:boto:Params: {} 2023-09-29T22:17:23.112 DEBUG:boto:Token: None 2023-09-29T22:17:23.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac55b0d02a424a15-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.158 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.159 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.159 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.159 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.159 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.159 DEBUG:boto:Method: GET 2023-09-29T22:17:23.160 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.160 DEBUG:boto:Data: 2023-09-29T22:17:23.160 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.160 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.160 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.160 DEBUG:boto:Params: {} 2023-09-29T22:17:23.161 DEBUG:boto:Token: None 2023-09-29T22:17:23.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.161 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000735378450cae132-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.163 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.164 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.164 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.164 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.164 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.164 DEBUG:boto:Data: 2023-09-29T22:17:23.164 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.165 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.165 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.165 DEBUG:boto:Params: {} 2023-09-29T22:17:23.165 DEBUG:boto:Token: None 2023-09-29T22:17:23.165 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg= 2023-09-29T22:17:23.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg=', 'Content-Length': '0'} 2023-09-29T22:17:23.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000161e60c2f04c19b4-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.167 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.168 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.168 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.168 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.168 DEBUG:boto:Data: 2023-09-29T22:17:23.168 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.168 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.168 DEBUG:boto:Params: {} 2023-09-29T22:17:23.168 DEBUG:boto:Token: None 2023-09-29T22:17:23.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg= 2023-09-29T22:17:23.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg=', 'Content-Length': '0'} 2023-09-29T22:17:23.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d80ba97c6b3d2fa-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.171 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.171 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.171 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.171 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.171 DEBUG:boto:Method: GET 2023-09-29T22:17:23.171 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.172 DEBUG:boto:Data: 2023-09-29T22:17:23.172 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.172 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.172 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.172 DEBUG:boto:Params: {} 2023-09-29T22:17:23.172 DEBUG:boto:Token: None 2023-09-29T22:17:23.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c16c552358e4dc01-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.218 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.219 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.219 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.219 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.219 DEBUG:boto:Method: GET 2023-09-29T22:17:23.219 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.219 DEBUG:boto:Data: 2023-09-29T22:17:23.219 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.219 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.219 DEBUG:boto:Params: {} 2023-09-29T22:17:23.220 DEBUG:boto:Token: None 2023-09-29T22:17:23.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000156159bbff0f7bec-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.222 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.222 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.222 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.222 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.222 DEBUG:boto:Method: GET 2023-09-29T22:17:23.222 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.223 DEBUG:boto:Data: 2023-09-29T22:17:23.223 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.223 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.223 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.223 DEBUG:boto:Params: {} 2023-09-29T22:17:23.223 DEBUG:boto:Token: None 2023-09-29T22:17:23.223 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4262f8c7d6159ad-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.266 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.267 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.267 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.267 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.267 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.267 DEBUG:boto:Method: GET 2023-09-29T22:17:23.267 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.267 DEBUG:boto:Data: 2023-09-29T22:17:23.268 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.268 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.268 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.268 DEBUG:boto:Params: {} 2023-09-29T22:17:23.268 DEBUG:boto:Token: None 2023-09-29T22:17:23.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042778c876378fecb-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.271 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:23.271 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.271 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.271 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.272 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.272 DEBUG:boto:Method: GET 2023-09-29T22:17:23.272 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.272 DEBUG:boto:Data: 2023-09-29T22:17:23.272 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.272 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.272 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.272 DEBUG:boto:Params: {} 2023-09-29T22:17:23.272 DEBUG:boto:Token: None 2023-09-29T22:17:23.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.273 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0c0484f7c9857f8-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.275 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.275 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.275 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.275 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.275 DEBUG:boto:Method: GET 2023-09-29T22:17:23.275 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.275 DEBUG:boto:Data: 2023-09-29T22:17:23.275 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.276 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.276 DEBUG:boto:Params: {} 2023-09-29T22:17:23.276 DEBUG:boto:Token: None 2023-09-29T22:17:23.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae109f26d241b602-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.278 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.278 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.278 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.278 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.279 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.279 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.279 DEBUG:boto:Data: 2023-09-29T22:17:23.279 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.279 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.279 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.279 DEBUG:boto:Params: {} 2023-09-29T22:17:23.279 DEBUG:boto:Token: None 2023-09-29T22:17:23.279 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.280 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CicWw4smn89nb9l6HfVdgM6+y5w= 2023-09-29T22:17:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CicWw4smn89nb9l6HfVdgM6+y5w=', 'Content-Length': '0'} 2023-09-29T22:17:23.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bf3b6d04b00eeaf-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.281 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.282 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.282 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.282 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.282 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.282 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.282 DEBUG:boto:Data: 2023-09-29T22:17:23.282 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.282 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.282 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.283 DEBUG:boto:Params: {} 2023-09-29T22:17:23.283 DEBUG:boto:Token: None 2023-09-29T22:17:23.283 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.283 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CicWw4smn89nb9l6HfVdgM6+y5w= 2023-09-29T22:17:23.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CicWw4smn89nb9l6HfVdgM6+y5w=', 'Content-Length': '0'} 2023-09-29T22:17:23.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9cf636f55d89f1b-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:23.285 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.285 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.285 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.285 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.285 DEBUG:boto:Method: GET 2023-09-29T22:17:23.286 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.286 DEBUG:boto:Data: 2023-09-29T22:17:23.286 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.286 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.286 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.286 DEBUG:boto:Params: {} 2023-09-29T22:17:23.286 DEBUG:boto:Token: None 2023-09-29T22:17:23.286 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.286 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e230e0f26545f8b-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.289 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.289 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.289 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.289 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.289 DEBUG:boto:Method: GET 2023-09-29T22:17:23.289 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.289 DEBUG:boto:Data: 2023-09-29T22:17:23.289 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.290 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.290 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.290 DEBUG:boto:Params: {} 2023-09-29T22:17:23.290 DEBUG:boto:Token: None 2023-09-29T22:17:23.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.290 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000356dcc7e21625085-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.334 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.334 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.335 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.335 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.335 DEBUG:boto:Method: GET 2023-09-29T22:17:23.335 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.335 DEBUG:boto:Data: 2023-09-29T22:17:23.335 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.335 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.335 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.335 DEBUG:boto:Params: {} 2023-09-29T22:17:23.335 DEBUG:boto:Token: None 2023-09-29T22:17:23.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fcfe7a5b5c0134-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.338 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.338 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.338 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.339 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.339 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.339 DEBUG:boto:Method: GET 2023-09-29T22:17:23.339 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.339 DEBUG:boto:Data: 2023-09-29T22:17:23.339 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.339 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.339 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.339 DEBUG:boto:Params: {} 2023-09-29T22:17:23.339 DEBUG:boto:Token: None 2023-09-29T22:17:23.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2be1faed5593cf-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.382 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.383 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.383 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.383 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.383 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.383 DEBUG:boto:Data: 2023-09-29T22:17:23.383 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.383 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.383 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.383 DEBUG:boto:Params: {} 2023-09-29T22:17:23.384 DEBUG:boto:Token: None 2023-09-29T22:17:23.384 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.384 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4i/zDsRfLQl7C6bzeYcbGx/xKVI= 2023-09-29T22:17:23.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4i/zDsRfLQl7C6bzeYcbGx/xKVI=', 'Content-Length': '0'} 2023-09-29T22:17:23.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000707eff778171baad-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.386 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.386 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.387 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.387 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.387 DEBUG:boto:Data: 2023-09-29T22:17:23.387 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.387 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.387 DEBUG:boto:Params: {} 2023-09-29T22:17:23.387 DEBUG:boto:Token: None 2023-09-29T22:17:23.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.387 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4i/zDsRfLQl7C6bzeYcbGx/xKVI= 2023-09-29T22:17:23.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4i/zDsRfLQl7C6bzeYcbGx/xKVI=', 'Content-Length': '0'} 2023-09-29T22:17:23.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c503162c0740474a-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:23.390 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.390 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.390 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.390 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.390 DEBUG:boto:Method: GET 2023-09-29T22:17:23.391 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.391 DEBUG:boto:Data: 2023-09-29T22:17:23.391 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.391 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.391 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.391 DEBUG:boto:Params: {} 2023-09-29T22:17:23.391 DEBUG:boto:Token: None 2023-09-29T22:17:23.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5514ef6e4ebad83-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.393 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.393 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:23.394 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.394 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.394 DEBUG:boto:Method: GET 2023-09-29T22:17:23.394 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.394 DEBUG:boto:Data: 2023-09-29T22:17:23.394 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.394 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.394 DEBUG:boto:Params: {} 2023-09-29T22:17:23.394 DEBUG:boto:Token: None 2023-09-29T22:17:23.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:23.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w= 2023-09-29T22:17:23.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ssi9UEpg4/x18x84VRCJIxwQY7w=', 'Content-Length': '0'} 2023-09-29T22:17:23.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076518a84ed75db0d-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.438 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.439 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.439 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.439 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.439 DEBUG:boto:Method: GET 2023-09-29T22:17:23.439 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.439 DEBUG:boto:Data: 2023-09-29T22:17:23.439 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.439 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.439 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.440 DEBUG:boto:Params: {} 2023-09-29T22:17:23.440 DEBUG:boto:Token: None 2023-09-29T22:17:23.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35d1cb5de9de1b8-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.442 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.443 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.443 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.443 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.443 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.443 DEBUG:boto:Method: GET 2023-09-29T22:17:23.443 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:23.444 DEBUG:boto:Data: 2023-09-29T22:17:23.444 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.444 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.444 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.444 DEBUG:boto:Params: {} 2023-09-29T22:17:23.444 DEBUG:boto:Token: None 2023-09-29T22:17:23.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.444 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006639621869ca95ae-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.446 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:23.447 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.447 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.447 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.447 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.447 DEBUG:boto:Method: GET 2023-09-29T22:17:23.448 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.448 DEBUG:boto:Data: 2023-09-29T22:17:23.448 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.448 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.448 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.448 DEBUG:boto:Params: {} 2023-09-29T22:17:23.448 DEBUG:boto:Token: None 2023-09-29T22:17:23.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e67497024702cdb-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.494 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.495 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.495 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.495 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.495 DEBUG:boto:Method: GET 2023-09-29T22:17:23.495 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.495 DEBUG:boto:Data: 2023-09-29T22:17:23.495 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.496 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.496 DEBUG:boto:Params: {} 2023-09-29T22:17:23.496 DEBUG:boto:Token: None 2023-09-29T22:17:23.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f674e2570da2933-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.538 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.538 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.539 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.539 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.539 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.539 DEBUG:boto:Data: 2023-09-29T22:17:23.539 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.539 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.539 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.539 DEBUG:boto:Params: {} 2023-09-29T22:17:23.540 DEBUG:boto:Token: None 2023-09-29T22:17:23.540 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CzHs4a//cARZSZZ8dc17noXY5/0= 2023-09-29T22:17:23.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CzHs4a//cARZSZZ8dc17noXY5/0=', 'Content-Length': '0'} 2023-09-29T22:17:23.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9b7789591c12dfe-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.542 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.542 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.542 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.542 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.543 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.543 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.543 DEBUG:boto:Data: 2023-09-29T22:17:23.543 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.543 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.543 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.543 DEBUG:boto:Params: {} 2023-09-29T22:17:23.544 DEBUG:boto:Token: None 2023-09-29T22:17:23.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CzHs4a//cARZSZZ8dc17noXY5/0= 2023-09-29T22:17:23.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CzHs4a//cARZSZZ8dc17noXY5/0=', 'Content-Length': '0'} 2023-09-29T22:17:23.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f25542112d77969e-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:23.546 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.546 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.546 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.547 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.547 DEBUG:boto:Method: GET 2023-09-29T22:17:23.547 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.547 DEBUG:boto:Data: 2023-09-29T22:17:23.547 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.547 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.547 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.547 DEBUG:boto:Params: {} 2023-09-29T22:17:23.548 DEBUG:boto:Token: None 2023-09-29T22:17:23.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.548 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089b626fb049c0e5a-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.594 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.594 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.595 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.595 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.595 DEBUG:boto:Method: GET 2023-09-29T22:17:23.595 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.595 DEBUG:boto:Data: 2023-09-29T22:17:23.595 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.595 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.595 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.595 DEBUG:boto:Params: {} 2023-09-29T22:17:23.596 DEBUG:boto:Token: None 2023-09-29T22:17:23.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1f14599865f024b-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.638 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.638 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.639 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.639 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.639 DEBUG:boto:Method: GET 2023-09-29T22:17:23.639 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.639 DEBUG:boto:Data: 2023-09-29T22:17:23.639 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.639 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.639 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.639 DEBUG:boto:Params: {} 2023-09-29T22:17:23.640 DEBUG:boto:Token: None 2023-09-29T22:17:23.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f564b543f7e1ff80-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.642 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.643 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.643 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.643 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.643 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.643 DEBUG:boto:Method: GET 2023-09-29T22:17:23.643 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.643 DEBUG:boto:Data: 2023-09-29T22:17:23.644 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.644 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.644 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.644 DEBUG:boto:Params: {} 2023-09-29T22:17:23.644 DEBUG:boto:Token: None 2023-09-29T22:17:23.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009130191594bfa12e-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.646 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.647 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.647 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.647 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.647 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.647 DEBUG:boto:Data: 2023-09-29T22:17:23.647 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.647 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.647 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.647 DEBUG:boto:Params: {} 2023-09-29T22:17:23.648 DEBUG:boto:Token: None 2023-09-29T22:17:23.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.648 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O0Q1GdXXPAPW6dxpbNIOKDry+4g= 2023-09-29T22:17:23.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O0Q1GdXXPAPW6dxpbNIOKDry+4g=', 'Content-Length': '0'} 2023-09-29T22:17:23.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0510fbf5281026f-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.650 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.650 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.651 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.651 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.651 DEBUG:boto:Data: 2023-09-29T22:17:23.651 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.651 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.651 DEBUG:boto:Params: {} 2023-09-29T22:17:23.651 DEBUG:boto:Token: None 2023-09-29T22:17:23.651 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O0Q1GdXXPAPW6dxpbNIOKDry+4g= 2023-09-29T22:17:23.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O0Q1GdXXPAPW6dxpbNIOKDry+4g=', 'Content-Length': '0'} 2023-09-29T22:17:23.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3bc4597ac0c6006-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:23.654 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.654 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.654 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.654 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.654 DEBUG:boto:Method: GET 2023-09-29T22:17:23.654 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.654 DEBUG:boto:Data: 2023-09-29T22:17:23.655 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.655 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.655 DEBUG:boto:Params: {} 2023-09-29T22:17:23.655 DEBUG:boto:Token: None 2023-09-29T22:17:23.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4af43f58a739fb3-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.698 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.699 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:23.699 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.699 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.699 DEBUG:boto:Method: GET 2023-09-29T22:17:23.699 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.699 DEBUG:boto:Data: 2023-09-29T22:17:23.699 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.700 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.700 DEBUG:boto:Params: {} 2023-09-29T22:17:23.700 DEBUG:boto:Token: None 2023-09-29T22:17:23.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:23.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA= 2023-09-29T22:17:23.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s5oII3b9e7esKEQoFoFebf7PBCA=', 'Content-Length': '0'} 2023-09-29T22:17:23.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e7caf30fdb107ac-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.742 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.743 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.743 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.743 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.743 DEBUG:boto:Method: GET 2023-09-29T22:17:23.744 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.744 DEBUG:boto:Data: 2023-09-29T22:17:23.745 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.745 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:23.745 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.745 DEBUG:boto:Params: {} 2023-09-29T22:17:23.745 DEBUG:boto:Token: None 2023-09-29T22:17:23.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029baf20039e0a219-0065174cf3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.790 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.791 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.791 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.791 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.791 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.791 DEBUG:boto:Method: GET 2023-09-29T22:17:23.791 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:23.792 DEBUG:boto:Data: 2023-09-29T22:17:23.792 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.792 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.792 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.792 DEBUG:boto:Params: {} 2023-09-29T22:17:23.792 DEBUG:boto:Token: None 2023-09-29T22:17:23.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe2d66d2b4c1473-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.795 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.795 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone2, test-zone1} 2023-09-29T22:17:23.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone3, test-zone1} 2023-09-29T22:17:23.796 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.796 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.796 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.796 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:23.796 DEBUG:boto:Data: 2023-09-29T22:17:23.796 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.796 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.796 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.796 DEBUG:boto:Params: {} 2023-09-29T22:17:23.797 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:17:23.797 DEBUG:boto:Token: None 2023-09-29T22:17:23.797 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.797 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DSn9HsOqrs/fT/P3deP5QBpuVP8= 2023-09-29T22:17:23.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DSn9HsOqrs/fT/P3deP5QBpuVP8=', 'Content-Length': '0'} 2023-09-29T22:17:23.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009319f6844ea9723b-0065174cf3-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.801 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.801 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.801 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.801 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:23.801 DEBUG:boto:Data: 2023-09-29T22:17:23.801 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.802 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.802 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.802 DEBUG:boto:Params: {} 2023-09-29T22:17:23.802 DEBUG:boto:Token: None 2023-09-29T22:17:23.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DSn9HsOqrs/fT/P3deP5QBpuVP8= 2023-09-29T22:17:23.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DSn9HsOqrs/fT/P3deP5QBpuVP8=', 'Content-Length': '0'} 2023-09-29T22:17:23.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6bd69981fcd0cc0-0065174cf3-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.804 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:23.804 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.805 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.805 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.805 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.805 DEBUG:boto:Method: GET 2023-09-29T22:17:23.805 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.805 DEBUG:boto:Data: 2023-09-29T22:17:23.805 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.805 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.805 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.805 DEBUG:boto:Params: {} 2023-09-29T22:17:23.806 DEBUG:boto:Token: None 2023-09-29T22:17:23.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/?versions 2023-09-29T22:17:23.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU= 2023-09-29T22:17:23.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU=', 'Content-Length': '0'} 2023-09-29T22:17:23.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c3eab653c0bc86-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.808 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.809 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:23.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:23.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:23.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:23.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:23.809 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.809 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.810 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.810 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.810 DEBUG:boto:Method: GET 2023-09-29T22:17:23.810 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.810 DEBUG:boto:Data: 2023-09-29T22:17:23.810 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.810 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.810 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.810 DEBUG:boto:Params: {} 2023-09-29T22:17:23.810 DEBUG:boto:Token: None 2023-09-29T22:17:23.811 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/?versions 2023-09-29T22:17:23.811 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU= 2023-09-29T22:17:23.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU=', 'Content-Length': '0'} 2023-09-29T22:17:23.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc01204e6bdf56d1-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.813 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:23.814 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.814 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.814 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.815 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.815 DEBUG:boto:Method: GET 2023-09-29T22:17:23.815 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.815 DEBUG:boto:Data: 2023-09-29T22:17:23.815 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.815 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.815 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.815 DEBUG:boto:Params: {} 2023-09-29T22:17:23.815 DEBUG:boto:Token: None 2023-09-29T22:17:23.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/?versions 2023-09-29T22:17:23.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU= 2023-09-29T22:17:23.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU=', 'Content-Length': '0'} 2023-09-29T22:17:23.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60e86740f030176-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.818 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.818 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.818 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.819 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.819 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.819 DEBUG:boto:Method: GET 2023-09-29T22:17:23.819 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:23.819 DEBUG:boto:Data: 2023-09-29T22:17:23.819 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.819 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.821 DEBUG:boto:Params: {} 2023-09-29T22:17:23.821 DEBUG:boto:Token: None 2023-09-29T22:17:23.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/?versions 2023-09-29T22:17:23.821 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU= 2023-09-29T22:17:23.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jb9f9lfP+xPmc3SWJzodPcvtepU=', 'Content-Length': '0'} 2023-09-29T22:17:23.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009817f292380f7d66-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.824 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:23.824 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.824 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.824 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.825 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.825 DEBUG:boto:Method: GET 2023-09-29T22:17:23.825 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.825 DEBUG:boto:Data: 2023-09-29T22:17:23.825 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.825 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.825 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.825 DEBUG:boto:Params: {} 2023-09-29T22:17:23.825 DEBUG:boto:Token: None 2023-09-29T22:17:23.826 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.826 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5c5a14f4caf692f-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.827 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.828 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.828 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.828 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.828 DEBUG:boto:Method: GET 2023-09-29T22:17:23.828 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.828 DEBUG:boto:Data: 2023-09-29T22:17:23.828 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.828 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.828 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.828 DEBUG:boto:Params: {} 2023-09-29T22:17:23.829 DEBUG:boto:Token: None 2023-09-29T22:17:23.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e74afbfe3ac6b2a-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.831 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.831 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.831 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.831 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.831 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.831 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.831 DEBUG:boto:Data: 2023-09-29T22:17:23.831 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.832 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.832 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.832 DEBUG:boto:Params: {} 2023-09-29T22:17:23.832 DEBUG:boto:Token: None 2023-09-29T22:17:23.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ROn9D6zTQGLsQaDIToRDycPSNLs= 2023-09-29T22:17:23.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ROn9D6zTQGLsQaDIToRDycPSNLs=', 'Content-Length': '0'} 2023-09-29T22:17:23.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5aaa3a055123c05-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.834 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.834 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.834 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.834 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.834 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.834 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.835 DEBUG:boto:Data: 2023-09-29T22:17:23.835 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.835 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.835 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.835 DEBUG:boto:Params: {} 2023-09-29T22:17:23.835 DEBUG:boto:Token: None 2023-09-29T22:17:23.835 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ROn9D6zTQGLsQaDIToRDycPSNLs= 2023-09-29T22:17:23.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ROn9D6zTQGLsQaDIToRDycPSNLs=', 'Content-Length': '0'} 2023-09-29T22:17:23.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087653b047078b339-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:23.837 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.837 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.837 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.837 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.837 DEBUG:boto:Method: GET 2023-09-29T22:17:23.838 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.838 DEBUG:boto:Data: 2023-09-29T22:17:23.838 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.838 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.838 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.838 DEBUG:boto:Params: {} 2023-09-29T22:17:23.838 DEBUG:boto:Token: None 2023-09-29T22:17:23.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.838 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004be22892aa25d1d-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.840 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.840 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.840 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.840 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.841 DEBUG:boto:Method: GET 2023-09-29T22:17:23.841 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.841 DEBUG:boto:Data: 2023-09-29T22:17:23.841 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.841 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.841 DEBUG:boto:Params: {} 2023-09-29T22:17:23.841 DEBUG:boto:Token: None 2023-09-29T22:17:23.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db8fa959fb3373b7-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.843 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.843 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.844 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.844 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.844 DEBUG:boto:Method: GET 2023-09-29T22:17:23.844 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.844 DEBUG:boto:Data: 2023-09-29T22:17:23.844 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.844 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.844 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.844 DEBUG:boto:Params: {} 2023-09-29T22:17:23.844 DEBUG:boto:Token: None 2023-09-29T22:17:23.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345f3d71dff9d1f8-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.847 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.847 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.847 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.848 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.848 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.848 DEBUG:boto:Method: GET 2023-09-29T22:17:23.848 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.848 DEBUG:boto:Data: 2023-09-29T22:17:23.848 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.848 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.848 DEBUG:boto:Params: {} 2023-09-29T22:17:23.848 DEBUG:boto:Token: None 2023-09-29T22:17:23.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.849 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0281ec8535cf5df-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.894 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.895 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.895 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.895 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.895 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:23.895 DEBUG:boto:Data: 2023-09-29T22:17:23.895 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.896 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.896 DEBUG:boto:Params: {} 2023-09-29T22:17:23.896 DEBUG:boto:Token: None 2023-09-29T22:17:23.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy 2023-09-29T22:17:23.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zy1yP7oj/wCn27SNzjOoCwxdBbc= 2023-09-29T22:17:23.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zy1yP7oj/wCn27SNzjOoCwxdBbc=', 'Content-Length': '0'} 2023-09-29T22:17:23.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000353c9e70e89a5991-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.898 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.899 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.899 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.899 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:23.899 DEBUG:boto:Data: 2023-09-29T22:17:23.899 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.899 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.899 DEBUG:boto:Params: {} 2023-09-29T22:17:23.899 DEBUG:boto:Token: None 2023-09-29T22:17:23.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy 2023-09-29T22:17:23.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zy1yP7oj/wCn27SNzjOoCwxdBbc= 2023-09-29T22:17:23.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zy1yP7oj/wCn27SNzjOoCwxdBbc=', 'Content-Length': '0'} 2023-09-29T22:17:23.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc34c9d2ea8d24d3-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:23.902 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.903 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.903 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.903 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.903 DEBUG:boto:Method: GET 2023-09-29T22:17:23.903 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.903 DEBUG:boto:Data: 2023-09-29T22:17:23.903 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.903 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.903 DEBUG:boto:Params: {} 2023-09-29T22:17:23.904 DEBUG:boto:Token: None 2023-09-29T22:17:23.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e542a43f5f537481-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.906 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:23.906 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:23.906 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.906 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.906 DEBUG:boto:Method: GET 2023-09-29T22:17:23.906 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.907 DEBUG:boto:Data: 2023-09-29T22:17:23.907 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.907 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.907 DEBUG:boto:Params: {} 2023-09-29T22:17:23.907 DEBUG:boto:Token: None 2023-09-29T22:17:23.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:23.907 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4= 2023-09-29T22:17:23.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rDg8kyrkK0MCT03aUy9ZgBUx9N4=', 'Content-Length': '0'} 2023-09-29T22:17:23.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000129717db30e24b05-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.909 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.909 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.909 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.910 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.910 DEBUG:boto:Method: GET 2023-09-29T22:17:23.910 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.910 DEBUG:boto:Data: 2023-09-29T22:17:23.910 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.910 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.910 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.910 DEBUG:boto:Params: {} 2023-09-29T22:17:23.910 DEBUG:boto:Token: None 2023-09-29T22:17:23.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.911 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb656ae85131950-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.913 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.913 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.914 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.914 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.914 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.914 DEBUG:boto:Method: GET 2023-09-29T22:17:23.914 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:23.914 DEBUG:boto:Data: 2023-09-29T22:17:23.914 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.914 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.914 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.915 DEBUG:boto:Params: {} 2023-09-29T22:17:23.915 DEBUG:boto:Token: None 2023-09-29T22:17:23.915 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.915 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338d1a2a1b2dc650-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.917 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.918 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.918 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.918 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.918 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.918 DEBUG:boto:Method: GET 2023-09-29T22:17:23.918 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.918 DEBUG:boto:Data: 2023-09-29T22:17:23.918 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.919 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.919 DEBUG:boto:Params: {} 2023-09-29T22:17:23.919 DEBUG:boto:Token: None 2023-09-29T22:17:23.919 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.919 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062713e07e77b71e2-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.921 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.921 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.921 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.921 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.921 DEBUG:boto:Method: GET 2023-09-29T22:17:23.922 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.922 DEBUG:boto:Data: 2023-09-29T22:17:23.922 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.922 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.922 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.922 DEBUG:boto:Params: {} 2023-09-29T22:17:23.922 DEBUG:boto:Token: None 2023-09-29T22:17:23.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.923 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef24db217ccd439e-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.924 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.924 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.925 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.925 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.925 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.925 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.925 DEBUG:boto:Data: 2023-09-29T22:17:23.925 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.925 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.925 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.925 DEBUG:boto:Params: {} 2023-09-29T22:17:23.926 DEBUG:boto:Token: None 2023-09-29T22:17:23.926 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.926 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0= 2023-09-29T22:17:23.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0=', 'Content-Length': '0'} 2023-09-29T22:17:23.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d9727afc702a401-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.928 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.928 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.928 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.928 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.928 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.928 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.929 DEBUG:boto:Data: 2023-09-29T22:17:23.929 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.929 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.929 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.929 DEBUG:boto:Params: {} 2023-09-29T22:17:23.929 DEBUG:boto:Token: None 2023-09-29T22:17:23.929 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.929 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0= 2023-09-29T22:17:23.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L0gEOUuaMHSXGeGvkWM+c+opsj0=', 'Content-Length': '0'} 2023-09-29T22:17:23.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090f26f42f505d296-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.932 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.932 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.932 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.932 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.932 DEBUG:boto:Method: GET 2023-09-29T22:17:23.932 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.932 DEBUG:boto:Data: 2023-09-29T22:17:23.932 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.932 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.932 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.932 DEBUG:boto:Params: {} 2023-09-29T22:17:23.933 DEBUG:boto:Token: None 2023-09-29T22:17:23.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c11a4a0d45816551-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.978 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.978 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.979 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.979 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.979 DEBUG:boto:Method: GET 2023-09-29T22:17:23.979 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.979 DEBUG:boto:Data: 2023-09-29T22:17:23.979 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.979 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.979 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.979 DEBUG:boto:Params: {} 2023-09-29T22:17:23.980 DEBUG:boto:Token: None 2023-09-29T22:17:23.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.980 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:23.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f60fbee72f5a957-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.982 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.982 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.982 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.982 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.982 DEBUG:boto:Method: GET 2023-09-29T22:17:23.982 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.982 DEBUG:boto:Data: 2023-09-29T22:17:23.983 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.983 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.983 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.983 DEBUG:boto:Params: {} 2023-09-29T22:17:23.983 DEBUG:boto:Token: None 2023-09-29T22:17:23.983 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.983 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0f17a22976c605-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.985 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.986 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:23.986 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:23.986 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.986 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.986 DEBUG:boto:Method: GET 2023-09-29T22:17:23.986 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:23.986 DEBUG:boto:Data: 2023-09-29T22:17:23.987 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.987 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.987 DEBUG:boto:Params: {} 2023-09-29T22:17:23.987 DEBUG:boto:Token: None 2023-09-29T22:17:23.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/ 2023-09-29T22:17:23.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw= 2023-09-29T22:17:23.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KCfSLhGRB+ISVN2XK5DGo780Mvw=', 'Content-Length': '0'} 2023-09-29T22:17:23.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5973c5021b0ec3-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.989 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:23.990 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.990 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.990 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.990 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.990 DEBUG:boto:Data: 2023-09-29T22:17:23.990 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.990 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.991 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.991 DEBUG:boto:Params: {} 2023-09-29T22:17:23.991 DEBUG:boto:Token: None 2023-09-29T22:17:23.991 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.991 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg= 2023-09-29T22:17:23.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg=', 'Content-Length': '0'} 2023-09-29T22:17:23.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2258fcc9773ea1f-0065174cf3-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.993 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.993 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.993 DEBUG:boto:Method: HEAD 2023-09-29T22:17:23.993 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.994 DEBUG:boto:Data: 2023-09-29T22:17:23.994 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.994 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:23.994 DEBUG:boto:Port: 8000 2023-09-29T22:17:23.994 DEBUG:boto:Params: {} 2023-09-29T22:17:23.994 DEBUG:boto:Token: None 2023-09-29T22:17:23.994 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.994 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg= 2023-09-29T22:17:23.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GBjvhSm0LgtiOZFLmK1kvf/8ceg=', 'Content-Length': '0'} 2023-09-29T22:17:23.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6e76a8f2a6fb951-0065174cf3-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:23.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:23.997 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.997 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:23.997 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.997 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.997 DEBUG:boto:Method: GET 2023-09-29T22:17:23.998 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.998 DEBUG:boto:Data: 2023-09-29T22:17:23.998 DEBUG:boto:Headers: {} 2023-09-29T22:17:23.998 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:23.998 DEBUG:boto:Port: 8001 2023-09-29T22:17:23.998 DEBUG:boto:Params: {} 2023-09-29T22:17:23.998 DEBUG:boto:Token: None 2023-09-29T22:17:23.998 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:23 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:23.998 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg= 2023-09-29T22:17:23.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VWm84FdY1+4W+kou2EL8bNcwuTg=', 'Content-Length': '0'} 2023-09-29T22:17:24.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2677a7ce599d142-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.042 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.043 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.043 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.043 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.043 DEBUG:boto:Method: GET 2023-09-29T22:17:24.043 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.043 DEBUG:boto:Data: 2023-09-29T22:17:24.043 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.043 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.044 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.044 DEBUG:boto:Params: {} 2023-09-29T22:17:24.044 DEBUG:boto:Token: None 2023-09-29T22:17:24.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw= 2023-09-29T22:17:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw=', 'Content-Length': '0'} 2023-09-29T22:17:24.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9bb2fcf99965fe4-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.046 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.046 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.046 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.047 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.047 DEBUG:boto:Method: GET 2023-09-29T22:17:24.047 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.047 DEBUG:boto:Data: 2023-09-29T22:17:24.047 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.047 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.047 DEBUG:boto:Params: {} 2023-09-29T22:17:24.047 DEBUG:boto:Token: None 2023-09-29T22:17:24.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc2f462f16986d0b-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.051 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.052 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.052 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.052 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.052 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.052 DEBUG:boto:Method: GET 2023-09-29T22:17:24.052 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.053 DEBUG:boto:Data: 2023-09-29T22:17:24.053 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.053 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.053 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.053 DEBUG:boto:Params: {} 2023-09-29T22:17:24.053 DEBUG:boto:Token: None 2023-09-29T22:17:24.053 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.053 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c927d0af8568e33-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.055 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:24.056 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.056 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.056 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.056 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.056 DEBUG:boto:Method: GET 2023-09-29T22:17:24.057 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.057 DEBUG:boto:Data: 2023-09-29T22:17:24.057 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.057 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.057 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.057 DEBUG:boto:Params: {} 2023-09-29T22:17:24.057 DEBUG:boto:Token: None 2023-09-29T22:17:24.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.057 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003544ffbdb4d0e0dc-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.102 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.103 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.103 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.103 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.103 DEBUG:boto:Method: GET 2023-09-29T22:17:24.103 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.103 DEBUG:boto:Data: 2023-09-29T22:17:24.103 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.103 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.104 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.104 DEBUG:boto:Params: {} 2023-09-29T22:17:24.104 DEBUG:boto:Token: None 2023-09-29T22:17:24.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001946aeaff9d2ea73-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.106 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.106 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.107 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.107 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.107 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.107 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.107 DEBUG:boto:Data: 2023-09-29T22:17:24.107 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.107 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.107 DEBUG:boto:Params: {} 2023-09-29T22:17:24.108 DEBUG:boto:Token: None 2023-09-29T22:17:24.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fP3OiFnXcoEQdOx9461/kS1AeN8= 2023-09-29T22:17:24.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fP3OiFnXcoEQdOx9461/kS1AeN8=', 'Content-Length': '0'} 2023-09-29T22:17:24.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae09303a02585e78-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.110 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.110 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.110 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.110 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.110 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.111 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.111 DEBUG:boto:Data: 2023-09-29T22:17:24.111 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.111 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.111 DEBUG:boto:Params: {} 2023-09-29T22:17:24.111 DEBUG:boto:Token: None 2023-09-29T22:17:24.111 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fP3OiFnXcoEQdOx9461/kS1AeN8= 2023-09-29T22:17:24.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fP3OiFnXcoEQdOx9461/kS1AeN8=', 'Content-Length': '0'} 2023-09-29T22:17:24.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a322b05be8488178-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:24.114 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.114 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.114 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.114 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.114 DEBUG:boto:Method: GET 2023-09-29T22:17:24.114 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.114 DEBUG:boto:Data: 2023-09-29T22:17:24.114 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.115 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.115 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.115 DEBUG:boto:Params: {} 2023-09-29T22:17:24.115 DEBUG:boto:Token: None 2023-09-29T22:17:24.115 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.115 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000395b50224e678f9a-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.162 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.163 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.163 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.163 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.163 DEBUG:boto:Method: GET 2023-09-29T22:17:24.163 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.163 DEBUG:boto:Data: 2023-09-29T22:17:24.163 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.163 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.163 DEBUG:boto:Params: {} 2023-09-29T22:17:24.164 DEBUG:boto:Token: None 2023-09-29T22:17:24.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.164 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000181e62628521325c-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.210 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.211 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.211 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.211 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.211 DEBUG:boto:Method: GET 2023-09-29T22:17:24.211 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.211 DEBUG:boto:Data: 2023-09-29T22:17:24.211 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.211 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.211 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.211 DEBUG:boto:Params: {} 2023-09-29T22:17:24.212 DEBUG:boto:Token: None 2023-09-29T22:17:24.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb98809544190b4-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.214 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.215 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.215 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.215 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.215 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.215 DEBUG:boto:Method: GET 2023-09-29T22:17:24.215 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.216 DEBUG:boto:Data: 2023-09-29T22:17:24.216 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.216 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.216 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.216 DEBUG:boto:Params: {} 2023-09-29T22:17:24.216 DEBUG:boto:Token: None 2023-09-29T22:17:24.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.216 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000324bfd386c8bd2b8-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.218 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.219 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.219 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.219 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.219 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.219 DEBUG:boto:Data: 2023-09-29T22:17:24.219 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.219 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.220 DEBUG:boto:Params: {} 2023-09-29T22:17:24.220 DEBUG:boto:Token: None 2023-09-29T22:17:24.220 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:d7DYmJW9KuJqBqGsi3DtK2OPWjs= 2023-09-29T22:17:24.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d7DYmJW9KuJqBqGsi3DtK2OPWjs=', 'Content-Length': '0'} 2023-09-29T22:17:24.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db6deda9ae6822b5-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.222 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.222 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.222 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.222 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.222 DEBUG:boto:Data: 2023-09-29T22:17:24.223 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.223 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.223 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.223 DEBUG:boto:Params: {} 2023-09-29T22:17:24.223 DEBUG:boto:Token: None 2023-09-29T22:17:24.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:d7DYmJW9KuJqBqGsi3DtK2OPWjs= 2023-09-29T22:17:24.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d7DYmJW9KuJqBqGsi3DtK2OPWjs=', 'Content-Length': '0'} 2023-09-29T22:17:24.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007857305166ef6fbd-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:24.225 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.226 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.226 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.226 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.226 DEBUG:boto:Method: GET 2023-09-29T22:17:24.226 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.226 DEBUG:boto:Data: 2023-09-29T22:17:24.226 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.226 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.226 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.227 DEBUG:boto:Params: {} 2023-09-29T22:17:24.227 DEBUG:boto:Token: None 2023-09-29T22:17:24.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.227 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000212e5ea9d1d8bed3-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.274 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.275 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:24.275 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.275 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.275 DEBUG:boto:Method: GET 2023-09-29T22:17:24.275 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.275 DEBUG:boto:Data: 2023-09-29T22:17:24.275 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.275 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.276 DEBUG:boto:Params: {} 2023-09-29T22:17:24.276 DEBUG:boto:Token: None 2023-09-29T22:17:24.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:24.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI= 2023-09-29T22:17:24.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWQ/T+MQYNegu1SgUlDF8UiaMoI=', 'Content-Length': '0'} 2023-09-29T22:17:24.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c00b97e9721979d3-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.322 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.323 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.323 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.323 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.323 DEBUG:boto:Method: GET 2023-09-29T22:17:24.323 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.323 DEBUG:boto:Data: 2023-09-29T22:17:24.323 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.323 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.324 DEBUG:boto:Params: {} 2023-09-29T22:17:24.324 DEBUG:boto:Token: None 2023-09-29T22:17:24.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eca761440f36321-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.327 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.327 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.327 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.327 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.327 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.327 DEBUG:boto:Method: GET 2023-09-29T22:17:24.328 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:24.328 DEBUG:boto:Data: 2023-09-29T22:17:24.328 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.328 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.328 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.328 DEBUG:boto:Params: {} 2023-09-29T22:17:24.328 DEBUG:boto:Token: None 2023-09-29T22:17:24.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.329 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f911f68e342331-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.331 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:24.331 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.332 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.332 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.332 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.332 DEBUG:boto:Method: GET 2023-09-29T22:17:24.332 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.332 DEBUG:boto:Data: 2023-09-29T22:17:24.332 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.332 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.332 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.333 DEBUG:boto:Params: {} 2023-09-29T22:17:24.333 DEBUG:boto:Token: None 2023-09-29T22:17:24.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ae8604a30745f79-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.335 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.335 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.335 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.335 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.336 DEBUG:boto:Method: GET 2023-09-29T22:17:24.336 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.336 DEBUG:boto:Data: 2023-09-29T22:17:24.336 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.336 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.336 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.336 DEBUG:boto:Params: {} 2023-09-29T22:17:24.336 DEBUG:boto:Token: None 2023-09-29T22:17:24.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.337 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009054b72a923a115f-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.339 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.339 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.339 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.339 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.339 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.339 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.339 DEBUG:boto:Data: 2023-09-29T22:17:24.340 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.340 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.340 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.340 DEBUG:boto:Params: {} 2023-09-29T22:17:24.340 DEBUG:boto:Token: None 2023-09-29T22:17:24.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fsmv7LLgC4b4ozVzj0JXt5u8jro= 2023-09-29T22:17:24.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fsmv7LLgC4b4ozVzj0JXt5u8jro=', 'Content-Length': '0'} 2023-09-29T22:17:24.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000486586d3e158c623-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.342 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.342 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.342 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.343 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.343 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.343 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.343 DEBUG:boto:Data: 2023-09-29T22:17:24.343 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.343 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.343 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.343 DEBUG:boto:Params: {} 2023-09-29T22:17:24.343 DEBUG:boto:Token: None 2023-09-29T22:17:24.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fsmv7LLgC4b4ozVzj0JXt5u8jro= 2023-09-29T22:17:24.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fsmv7LLgC4b4ozVzj0JXt5u8jro=', 'Content-Length': '0'} 2023-09-29T22:17:24.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b23922ba414c3d69-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:24.345 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.346 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.346 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.346 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.346 DEBUG:boto:Method: GET 2023-09-29T22:17:24.346 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.346 DEBUG:boto:Data: 2023-09-29T22:17:24.346 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.346 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.346 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.347 DEBUG:boto:Params: {} 2023-09-29T22:17:24.347 DEBUG:boto:Token: None 2023-09-29T22:17:24.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000026db5d5d7c3145a-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.349 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.349 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.349 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.349 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.350 DEBUG:boto:Method: GET 2023-09-29T22:17:24.350 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.350 DEBUG:boto:Data: 2023-09-29T22:17:24.350 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.350 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.350 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.350 DEBUG:boto:Params: {} 2023-09-29T22:17:24.350 DEBUG:boto:Token: None 2023-09-29T22:17:24.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.351 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2f0f747283427f8-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.398 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.399 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.399 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.399 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.399 DEBUG:boto:Method: GET 2023-09-29T22:17:24.399 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.399 DEBUG:boto:Data: 2023-09-29T22:17:24.400 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.400 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.400 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.400 DEBUG:boto:Params: {} 2023-09-29T22:17:24.400 DEBUG:boto:Token: None 2023-09-29T22:17:24.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cce362b1955b356-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.403 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.403 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.403 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.403 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.403 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.403 DEBUG:boto:Method: GET 2023-09-29T22:17:24.404 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.404 DEBUG:boto:Data: 2023-09-29T22:17:24.404 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.404 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.404 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.404 DEBUG:boto:Params: {} 2023-09-29T22:17:24.404 DEBUG:boto:Token: None 2023-09-29T22:17:24.404 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6de338070996397-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.407 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.407 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.407 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.407 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.407 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.408 DEBUG:boto:Data: 2023-09-29T22:17:24.408 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.408 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.408 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.408 DEBUG:boto:Params: {} 2023-09-29T22:17:24.408 DEBUG:boto:Token: None 2023-09-29T22:17:24.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.408 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W5nonhCVsFstPutEYTPmoz/qP/I= 2023-09-29T22:17:24.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W5nonhCVsFstPutEYTPmoz/qP/I=', 'Content-Length': '0'} 2023-09-29T22:17:24.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2dd32eda503a9ee-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.410 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.410 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.410 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.411 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.411 DEBUG:boto:Data: 2023-09-29T22:17:24.411 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.411 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.411 DEBUG:boto:Params: {} 2023-09-29T22:17:24.411 DEBUG:boto:Token: None 2023-09-29T22:17:24.411 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.411 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W5nonhCVsFstPutEYTPmoz/qP/I= 2023-09-29T22:17:24.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W5nonhCVsFstPutEYTPmoz/qP/I=', 'Content-Length': '0'} 2023-09-29T22:17:24.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000951ed63ae28c828e-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:24.414 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.414 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.414 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.414 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.414 DEBUG:boto:Method: GET 2023-09-29T22:17:24.414 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.414 DEBUG:boto:Data: 2023-09-29T22:17:24.414 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.414 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.414 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.415 DEBUG:boto:Params: {} 2023-09-29T22:17:24.415 DEBUG:boto:Token: None 2023-09-29T22:17:24.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d196d198c2f4e05-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.458 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.458 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:24.459 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.459 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.459 DEBUG:boto:Method: GET 2023-09-29T22:17:24.459 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.459 DEBUG:boto:Data: 2023-09-29T22:17:24.459 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.460 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.460 DEBUG:boto:Params: {} 2023-09-29T22:17:24.460 DEBUG:boto:Token: None 2023-09-29T22:17:24.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:24.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8= 2023-09-29T22:17:24.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+ssC2bRUWBokAr0c21KVSV+0xu8=', 'Content-Length': '0'} 2023-09-29T22:17:24.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003058b8005fce2688-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.506 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.506 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.507 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.507 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.507 DEBUG:boto:Method: GET 2023-09-29T22:17:24.507 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.507 DEBUG:boto:Data: 2023-09-29T22:17:24.507 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.507 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.507 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.507 DEBUG:boto:Params: {} 2023-09-29T22:17:24.508 DEBUG:boto:Token: None 2023-09-29T22:17:24.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf1cf89db1c6ae3-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.510 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.511 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.511 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.511 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.511 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.511 DEBUG:boto:Method: GET 2023-09-29T22:17:24.511 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:24.512 DEBUG:boto:Data: 2023-09-29T22:17:24.512 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.512 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:24.512 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.512 DEBUG:boto:Params: {} 2023-09-29T22:17:24.512 DEBUG:boto:Token: None 2023-09-29T22:17:24.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b30623e7e44bbe1-0065174cf4-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.558 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.559 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone3, test-zone1} 2023-09-29T22:17:24.559 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone3, test-zone2} 2023-09-29T22:17:24.559 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.559 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.559 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.559 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:24.560 DEBUG:boto:Data: 2023-09-29T22:17:24.560 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.560 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.560 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.560 DEBUG:boto:Params: {} 2023-09-29T22:17:24.560 DEBUG:boto:Token: None 2023-09-29T22:17:24.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.561 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sl1NzYVlOxb/6iph2lDmhYSc9JQ= 2023-09-29T22:17:24.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sl1NzYVlOxb/6iph2lDmhYSc9JQ=', 'Content-Length': '0'} 2023-09-29T22:17:24.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b40c4f18580dfe9-0065174cf4-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.563 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.563 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.563 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.563 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:24.564 DEBUG:boto:Data: 2023-09-29T22:17:24.564 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.564 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.564 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.564 DEBUG:boto:Params: {} 2023-09-29T22:17:24.564 DEBUG:boto:Token: None 2023-09-29T22:17:24.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sl1NzYVlOxb/6iph2lDmhYSc9JQ= 2023-09-29T22:17:24.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sl1NzYVlOxb/6iph2lDmhYSc9JQ=', 'Content-Length': '0'} 2023-09-29T22:17:24.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b81f668dca911919-0065174cf4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:24.567 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.567 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.568 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.568 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.568 DEBUG:boto:Method: GET 2023-09-29T22:17:24.568 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.568 DEBUG:boto:Data: 2023-09-29T22:17:24.568 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.568 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.568 DEBUG:boto:Params: {} 2023-09-29T22:17:24.568 DEBUG:boto:Token: None 2023-09-29T22:17:24.569 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/?versions 2023-09-29T22:17:24.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc= 2023-09-29T22:17:24.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc=', 'Content-Length': '0'} 2023-09-29T22:17:24.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004746fc25ed8da4b1-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.571 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.572 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:24.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:24.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:24.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:24.572 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:24.572 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.572 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.573 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.573 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.573 DEBUG:boto:Method: GET 2023-09-29T22:17:24.573 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.573 DEBUG:boto:Data: 2023-09-29T22:17:24.573 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.573 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.573 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.573 DEBUG:boto:Params: {} 2023-09-29T22:17:24.573 DEBUG:boto:Token: None 2023-09-29T22:17:24.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/?versions 2023-09-29T22:17:24.574 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc= 2023-09-29T22:17:24.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc=', 'Content-Length': '0'} 2023-09-29T22:17:24.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda1ff8d169ccba4-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.576 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.577 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:24.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:24.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:24.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:24.577 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.577 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.578 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.578 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.578 DEBUG:boto:Method: GET 2023-09-29T22:17:24.578 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.578 DEBUG:boto:Data: 2023-09-29T22:17:24.578 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.578 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.578 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.578 DEBUG:boto:Params: {} 2023-09-29T22:17:24.579 DEBUG:boto:Token: None 2023-09-29T22:17:24.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/?versions 2023-09-29T22:17:24.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc= 2023-09-29T22:17:24.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc=', 'Content-Length': '0'} 2023-09-29T22:17:24.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000835f552fc22ad10-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.581 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.582 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.582 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.582 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.582 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.582 DEBUG:boto:Method: GET 2023-09-29T22:17:24.582 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:17:24.582 DEBUG:boto:Data: 2023-09-29T22:17:24.583 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.583 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.583 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.583 DEBUG:boto:Params: {} 2023-09-29T22:17:24.583 DEBUG:boto:Token: None 2023-09-29T22:17:24.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/?versions 2023-09-29T22:17:24.583 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc= 2023-09-29T22:17:24.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0cn73IF8jVaHg2XxIiSb713oUmc=', 'Content-Length': '0'} 2023-09-29T22:17:24.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000421a66809e16ad0b-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.586 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:24.587 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.587 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.587 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.587 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.587 DEBUG:boto:Method: GET 2023-09-29T22:17:24.587 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.587 DEBUG:boto:Data: 2023-09-29T22:17:24.587 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.588 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.588 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.588 DEBUG:boto:Params: {} 2023-09-29T22:17:24.588 DEBUG:boto:Token: None 2023-09-29T22:17:24.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.589 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7bf7953187154a8-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.591 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.591 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.591 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.592 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.592 DEBUG:boto:Method: GET 2023-09-29T22:17:24.592 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.592 DEBUG:boto:Data: 2023-09-29T22:17:24.592 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.592 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.592 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.593 DEBUG:boto:Params: {} 2023-09-29T22:17:24.593 DEBUG:boto:Token: None 2023-09-29T22:17:24.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.593 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3dfa586d05c61a8-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.595 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.596 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.596 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.596 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.596 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.596 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.597 DEBUG:boto:Data: 2023-09-29T22:17:24.597 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.597 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.597 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.597 DEBUG:boto:Params: {} 2023-09-29T22:17:24.597 DEBUG:boto:Token: None 2023-09-29T22:17:24.598 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vafdM9iKIlX2GsisqTVE+MJyAx8= 2023-09-29T22:17:24.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vafdM9iKIlX2GsisqTVE+MJyAx8=', 'Content-Length': '0'} 2023-09-29T22:17:24.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034d942d8f403173b-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.600 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.600 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.600 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.601 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.601 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.601 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.601 DEBUG:boto:Data: 2023-09-29T22:17:24.601 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.601 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.601 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.602 DEBUG:boto:Params: {} 2023-09-29T22:17:24.602 DEBUG:boto:Token: None 2023-09-29T22:17:24.602 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vafdM9iKIlX2GsisqTVE+MJyAx8= 2023-09-29T22:17:24.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vafdM9iKIlX2GsisqTVE+MJyAx8=', 'Content-Length': '0'} 2023-09-29T22:17:24.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013286e65ba2e70c9-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:24.604 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.605 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.605 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.605 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.605 DEBUG:boto:Method: GET 2023-09-29T22:17:24.605 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.605 DEBUG:boto:Data: 2023-09-29T22:17:24.605 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.605 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.605 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.605 DEBUG:boto:Params: {} 2023-09-29T22:17:24.606 DEBUG:boto:Token: None 2023-09-29T22:17:24.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da69f75fdc3b1246-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.608 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.608 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.608 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.608 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.608 DEBUG:boto:Method: GET 2023-09-29T22:17:24.608 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.609 DEBUG:boto:Data: 2023-09-29T22:17:24.609 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.609 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.609 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.609 DEBUG:boto:Params: {} 2023-09-29T22:17:24.609 DEBUG:boto:Token: None 2023-09-29T22:17:24.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.609 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000761a3aba0fa3e926-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.611 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.612 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.612 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.612 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.612 DEBUG:boto:Method: GET 2023-09-29T22:17:24.612 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.613 DEBUG:boto:Data: 2023-09-29T22:17:24.613 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.613 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.613 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.614 DEBUG:boto:Params: {} 2023-09-29T22:17:24.614 DEBUG:boto:Token: None 2023-09-29T22:17:24.614 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.614 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049f49c758cc6ee9-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.658 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.659 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.659 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.659 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.659 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.659 DEBUG:boto:Method: GET 2023-09-29T22:17:24.659 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.660 DEBUG:boto:Data: 2023-09-29T22:17:24.660 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.660 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.660 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.660 DEBUG:boto:Params: {} 2023-09-29T22:17:24.660 DEBUG:boto:Token: None 2023-09-29T22:17:24.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.661 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e404513c1ed4eb5d-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.664 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.664 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.664 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.665 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.665 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:24.665 DEBUG:boto:Data: 2023-09-29T22:17:24.665 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.665 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.665 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.665 DEBUG:boto:Params: {} 2023-09-29T22:17:24.665 DEBUG:boto:Token: None 2023-09-29T22:17:24.665 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy 2023-09-29T22:17:24.666 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MP0tfcZX+fx2t8tSEGWAIm3/xL4= 2023-09-29T22:17:24.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MP0tfcZX+fx2t8tSEGWAIm3/xL4=', 'Content-Length': '0'} 2023-09-29T22:17:24.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076de7949b461af90-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.668 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.668 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.668 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.668 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:17:24.669 DEBUG:boto:Data: 2023-09-29T22:17:24.669 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.669 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.669 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.669 DEBUG:boto:Params: {} 2023-09-29T22:17:24.669 DEBUG:boto:Token: None 2023-09-29T22:17:24.669 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy 2023-09-29T22:17:24.669 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MP0tfcZX+fx2t8tSEGWAIm3/xL4= 2023-09-29T22:17:24.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MP0tfcZX+fx2t8tSEGWAIm3/xL4=', 'Content-Length': '0'} 2023-09-29T22:17:24.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093d0f03da5574d70-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:24.672 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.672 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.672 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.672 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.672 DEBUG:boto:Method: GET 2023-09-29T22:17:24.672 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.672 DEBUG:boto:Data: 2023-09-29T22:17:24.673 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.673 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.673 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.673 DEBUG:boto:Params: {} 2023-09-29T22:17:24.673 DEBUG:boto:Token: None 2023-09-29T22:17:24.673 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.673 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd5d84a8b7edf558-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.675 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:17:24.675 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:17:24.676 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.676 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.676 DEBUG:boto:Method: GET 2023-09-29T22:17:24.676 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.676 DEBUG:boto:Data: 2023-09-29T22:17:24.676 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.676 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.676 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.676 DEBUG:boto:Params: {} 2023-09-29T22:17:24.677 DEBUG:boto:Token: None 2023-09-29T22:17:24.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:17:24.677 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo= 2023-09-29T22:17:24.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:asdWc6KU2UoGSruTGBQxqEFl4Oo=', 'Content-Length': '0'} 2023-09-29T22:17:24.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d83a8adc761e759-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.679 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.679 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.679 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.679 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.679 DEBUG:boto:Method: GET 2023-09-29T22:17:24.680 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.680 DEBUG:boto:Data: 2023-09-29T22:17:24.680 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.680 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.680 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.680 DEBUG:boto:Params: {} 2023-09-29T22:17:24.680 DEBUG:boto:Token: None 2023-09-29T22:17:24.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.681 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daaf7ca00383570d-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.684 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.684 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.685 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.685 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.685 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.685 DEBUG:boto:Method: GET 2023-09-29T22:17:24.685 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:17:24.686 DEBUG:boto:Data: 2023-09-29T22:17:24.686 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.686 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.686 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.686 DEBUG:boto:Params: {} 2023-09-29T22:17:24.687 DEBUG:boto:Token: None 2023-09-29T22:17:24.687 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065963ffbc02de1a4-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.734 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:24.735 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.735 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.735 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.735 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.736 DEBUG:boto:Method: GET 2023-09-29T22:17:24.736 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.736 DEBUG:boto:Data: 2023-09-29T22:17:24.736 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.736 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.736 DEBUG:boto:Params: {} 2023-09-29T22:17:24.736 DEBUG:boto:Token: None 2023-09-29T22:17:24.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw= 2023-09-29T22:17:24.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw=', 'Content-Length': '0'} 2023-09-29T22:17:24.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000827f7fd7f748cdd6-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.739 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.739 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.739 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.739 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.739 DEBUG:boto:Method: GET 2023-09-29T22:17:24.739 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.739 DEBUG:boto:Data: 2023-09-29T22:17:24.739 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.740 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.740 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.740 DEBUG:boto:Params: {} 2023-09-29T22:17:24.740 DEBUG:boto:Token: None 2023-09-29T22:17:24.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw= 2023-09-29T22:17:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw=', 'Content-Length': '0'} 2023-09-29T22:17:24.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a735b81552bcadc5-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.786 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.787 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.787 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.787 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.787 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.787 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.787 DEBUG:boto:Data: 2023-09-29T22:17:24.787 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.787 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.787 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.787 DEBUG:boto:Params: {} 2023-09-29T22:17:24.788 DEBUG:boto:Token: None 2023-09-29T22:17:24.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yeW/E/BrWMdXLM06O4yvHxdmCao= 2023-09-29T22:17:24.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yeW/E/BrWMdXLM06O4yvHxdmCao=', 'Content-Length': '0'} 2023-09-29T22:17:24.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e55a128f63aee64-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.790 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.790 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.791 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.791 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.791 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.791 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.791 DEBUG:boto:Data: 2023-09-29T22:17:24.791 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.791 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.791 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.791 DEBUG:boto:Params: {} 2023-09-29T22:17:24.792 DEBUG:boto:Token: None 2023-09-29T22:17:24.792 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yeW/E/BrWMdXLM06O4yvHxdmCao= 2023-09-29T22:17:24.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yeW/E/BrWMdXLM06O4yvHxdmCao=', 'Content-Length': '0'} 2023-09-29T22:17:24.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000580092c83b486852-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:24.794 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.794 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.794 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.794 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.795 DEBUG:boto:Method: GET 2023-09-29T22:17:24.795 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.795 DEBUG:boto:Data: 2023-09-29T22:17:24.795 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.795 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.795 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.795 DEBUG:boto:Params: {} 2023-09-29T22:17:24.795 DEBUG:boto:Token: None 2023-09-29T22:17:24.795 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.795 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw= 2023-09-29T22:17:24.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw=', 'Content-Length': '0'} 2023-09-29T22:17:24.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088107524c0b5e128-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.842 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.843 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.843 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.843 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.843 DEBUG:boto:Method: GET 2023-09-29T22:17:24.843 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.843 DEBUG:boto:Data: 2023-09-29T22:17:24.843 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.843 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.844 DEBUG:boto:Params: {} 2023-09-29T22:17:24.844 DEBUG:boto:Token: None 2023-09-29T22:17:24.846 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:24.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw= 2023-09-29T22:17:24.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8X4gbwH7BJpBuXOC77klGI2NAuw=', 'Content-Length': '0'} 2023-09-29T22:17:24.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027a838588b763f0e-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.890 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.891 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.891 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.891 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.891 DEBUG:boto:Method: GET 2023-09-29T22:17:24.891 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.891 DEBUG:boto:Data: 2023-09-29T22:17:24.891 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.891 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.892 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.892 DEBUG:boto:Params: {} 2023-09-29T22:17:24.892 DEBUG:boto:Token: None 2023-09-29T22:17:24.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698a8199e25cf904-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.942 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.943 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:24.943 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:24.943 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.943 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.943 DEBUG:boto:Method: GET 2023-09-29T22:17:24.943 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:24.943 DEBUG:boto:Data: 2023-09-29T22:17:24.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.944 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.944 DEBUG:boto:Params: {} 2023-09-29T22:17:24.944 DEBUG:boto:Token: None 2023-09-29T22:17:24.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/ 2023-09-29T22:17:24.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A= 2023-09-29T22:17:24.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TFzpCLrOgEQQssqy6i5YmII7b9A=', 'Content-Length': '0'} 2023-09-29T22:17:24.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011f167b6b7dab24e-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.990 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:24.991 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.991 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.991 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.991 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.992 DEBUG:boto:Data: 2023-09-29T22:17:24.992 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.992 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:24.992 DEBUG:boto:Port: 8001 2023-09-29T22:17:24.992 DEBUG:boto:Params: {} 2023-09-29T22:17:24.993 DEBUG:boto:Token: None 2023-09-29T22:17:24.993 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.993 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lTdQ6L9s/k0IOQNcru1DaKwHx4A= 2023-09-29T22:17:24.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lTdQ6L9s/k0IOQNcru1DaKwHx4A=', 'Content-Length': '0'} 2023-09-29T22:17:24.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f751d7415ed28d41-0065174cf4-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.995 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.995 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.996 DEBUG:boto:Method: HEAD 2023-09-29T22:17:24.996 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.996 DEBUG:boto:Data: 2023-09-29T22:17:24.996 DEBUG:boto:Headers: {} 2023-09-29T22:17:24.996 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:24.996 DEBUG:boto:Port: 8000 2023-09-29T22:17:24.996 DEBUG:boto:Params: {} 2023-09-29T22:17:24.996 DEBUG:boto:Token: None 2023-09-29T22:17:24.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:24 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lTdQ6L9s/k0IOQNcru1DaKwHx4A= 2023-09-29T22:17:24.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lTdQ6L9s/k0IOQNcru1DaKwHx4A=', 'Content-Length': '0'} 2023-09-29T22:17:24.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e5840f9eec3170a-0065174cf4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:24.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:24.999 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.999 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:24.999 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.000 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.000 DEBUG:boto:Method: GET 2023-09-29T22:17:25.000 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.000 DEBUG:boto:Data: 2023-09-29T22:17:25.000 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.000 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.000 DEBUG:boto:Params: {} 2023-09-29T22:17:25.000 DEBUG:boto:Token: None 2023-09-29T22:17:25.001 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.001 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3L8r/7VqepNip7n8O2OkVs+7suQ= 2023-09-29T22:17:25.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3L8r/7VqepNip7n8O2OkVs+7suQ=', 'Content-Length': '0'} 2023-09-29T22:17:25.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025449f3ce352a682-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.046 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:25.047 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:25.047 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.047 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.047 DEBUG:boto:Method: GET 2023-09-29T22:17:25.047 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.047 DEBUG:boto:Data: 2023-09-29T22:17:25.047 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.047 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.047 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.047 DEBUG:boto:Params: {} 2023-09-29T22:17:25.048 DEBUG:boto:Token: None 2023-09-29T22:17:25.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:17:25.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3L8r/7VqepNip7n8O2OkVs+7suQ= 2023-09-29T22:17:25.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3L8r/7VqepNip7n8O2OkVs+7suQ=', 'Content-Length': '0'} 2023-09-29T22:17:25.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045fe05a0384fcea3-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.094 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.095 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.095 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.095 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.095 DEBUG:boto:Method: GET 2023-09-29T22:17:25.095 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.095 DEBUG:boto:Data: 2023-09-29T22:17:25.095 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.095 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.095 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.096 DEBUG:boto:Params: {} 2023-09-29T22:17:25.096 DEBUG:boto:Token: None 2023-09-29T22:17:25.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008deed7b53f6e5672-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.146 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.147 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.147 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.147 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.147 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.147 DEBUG:boto:Method: GET 2023-09-29T22:17:25.147 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:17:25.147 DEBUG:boto:Data: 2023-09-29T22:17:25.148 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.148 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.148 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.148 DEBUG:boto:Params: {} 2023-09-29T22:17:25.148 DEBUG:boto:Token: None 2023-09-29T22:17:25.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac1c0b9b1a0e6988-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.194 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:25.195 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.195 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.195 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.195 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.196 DEBUG:boto:Method: GET 2023-09-29T22:17:25.196 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.196 DEBUG:boto:Data: 2023-09-29T22:17:25.196 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.196 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.196 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.196 DEBUG:boto:Params: {} 2023-09-29T22:17:25.196 DEBUG:boto:Token: None 2023-09-29T22:17:25.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e8323266ce6442b-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.199 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.199 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.199 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.199 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.199 DEBUG:boto:Method: GET 2023-09-29T22:17:25.199 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.200 DEBUG:boto:Data: 2023-09-29T22:17:25.200 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.200 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.200 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.200 DEBUG:boto:Params: {} 2023-09-29T22:17:25.200 DEBUG:boto:Token: None 2023-09-29T22:17:25.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000124fdbc5a88bec05-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.202 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.203 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.203 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.203 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.203 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.203 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.203 DEBUG:boto:Data: 2023-09-29T22:17:25.203 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.203 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.203 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.203 DEBUG:boto:Params: {} 2023-09-29T22:17:25.204 DEBUG:boto:Token: None 2023-09-29T22:17:25.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J5UgyZBcuL/XkIb7pCeEh2qbNNI= 2023-09-29T22:17:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J5UgyZBcuL/XkIb7pCeEh2qbNNI=', 'Content-Length': '0'} 2023-09-29T22:17:25.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002baca74cf985735a-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.206 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.206 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.206 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.206 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.206 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.206 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.206 DEBUG:boto:Data: 2023-09-29T22:17:25.207 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.207 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.207 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.207 DEBUG:boto:Params: {} 2023-09-29T22:17:25.207 DEBUG:boto:Token: None 2023-09-29T22:17:25.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.207 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J5UgyZBcuL/XkIb7pCeEh2qbNNI= 2023-09-29T22:17:25.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J5UgyZBcuL/XkIb7pCeEh2qbNNI=', 'Content-Length': '0'} 2023-09-29T22:17:25.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb7904ed1e83c2a4-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:25.209 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.210 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.210 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.210 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.210 DEBUG:boto:Method: GET 2023-09-29T22:17:25.210 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.210 DEBUG:boto:Data: 2023-09-29T22:17:25.210 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.210 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.210 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.210 DEBUG:boto:Params: {} 2023-09-29T22:17:25.211 DEBUG:boto:Token: None 2023-09-29T22:17:25.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.211 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc96eb7fbe26683a-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.254 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.255 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.255 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.255 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.255 DEBUG:boto:Method: GET 2023-09-29T22:17:25.255 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.255 DEBUG:boto:Data: 2023-09-29T22:17:25.255 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.255 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.255 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.255 DEBUG:boto:Params: {} 2023-09-29T22:17:25.256 DEBUG:boto:Token: None 2023-09-29T22:17:25.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052ec1948c93105c1-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.258 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.259 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.259 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.259 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.259 DEBUG:boto:Method: GET 2023-09-29T22:17:25.259 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.259 DEBUG:boto:Data: 2023-09-29T22:17:25.259 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.259 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.259 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.260 DEBUG:boto:Params: {} 2023-09-29T22:17:25.260 DEBUG:boto:Token: None 2023-09-29T22:17:25.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d2bf0a92aef51c-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.306 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.307 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.307 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.307 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.307 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.307 DEBUG:boto:Method: GET 2023-09-29T22:17:25.307 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.308 DEBUG:boto:Data: 2023-09-29T22:17:25.308 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.308 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.308 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.308 DEBUG:boto:Params: {} 2023-09-29T22:17:25.308 DEBUG:boto:Token: None 2023-09-29T22:17:25.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f961edc33630a8-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.311 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.312 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.312 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.312 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.312 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.312 DEBUG:boto:Data: 2023-09-29T22:17:25.312 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.312 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.312 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.312 DEBUG:boto:Params: {} 2023-09-29T22:17:25.313 DEBUG:boto:Token: None 2023-09-29T22:17:25.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.313 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R55b5Bx2NapvXWVvbNJqtUXaQ60= 2023-09-29T22:17:25.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R55b5Bx2NapvXWVvbNJqtUXaQ60=', 'Content-Length': '0'} 2023-09-29T22:17:25.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf43e1ca262d077a-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.315 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.315 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.315 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.315 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.316 DEBUG:boto:Data: 2023-09-29T22:17:25.316 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.316 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.316 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.316 DEBUG:boto:Params: {} 2023-09-29T22:17:25.316 DEBUG:boto:Token: None 2023-09-29T22:17:25.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R55b5Bx2NapvXWVvbNJqtUXaQ60= 2023-09-29T22:17:25.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R55b5Bx2NapvXWVvbNJqtUXaQ60=', 'Content-Length': '0'} 2023-09-29T22:17:25.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019cbd540c30c43e6-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:25.319 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.319 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.319 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.319 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.319 DEBUG:boto:Method: GET 2023-09-29T22:17:25.319 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.319 DEBUG:boto:Data: 2023-09-29T22:17:25.319 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.320 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.320 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.320 DEBUG:boto:Params: {} 2023-09-29T22:17:25.320 DEBUG:boto:Token: None 2023-09-29T22:17:25.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000000b52fdcebdf778-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.366 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.366 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:25.367 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.367 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.367 DEBUG:boto:Method: GET 2023-09-29T22:17:25.367 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.367 DEBUG:boto:Data: 2023-09-29T22:17:25.367 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.367 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.367 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.367 DEBUG:boto:Params: {} 2023-09-29T22:17:25.368 DEBUG:boto:Token: None 2023-09-29T22:17:25.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:17:25.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE= 2023-09-29T22:17:25.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HON4r9kjMx4De4gwIcJO9O225HE=', 'Content-Length': '0'} 2023-09-29T22:17:25.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015bc1ed90c2a4dab-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.370 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.370 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.370 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.371 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.371 DEBUG:boto:Method: GET 2023-09-29T22:17:25.371 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.371 DEBUG:boto:Data: 2023-09-29T22:17:25.371 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.371 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.371 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.371 DEBUG:boto:Params: {} 2023-09-29T22:17:25.371 DEBUG:boto:Token: None 2023-09-29T22:17:25.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096857b1f50991b8b-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.414 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.415 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.415 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.415 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.415 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.415 DEBUG:boto:Method: GET 2023-09-29T22:17:25.415 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:17:25.415 DEBUG:boto:Data: 2023-09-29T22:17:25.416 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.416 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.416 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.416 DEBUG:boto:Params: {} 2023-09-29T22:17:25.416 DEBUG:boto:Token: None 2023-09-29T22:17:25.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c867205ab36770e-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.419 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:25.420 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.420 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.420 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.420 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.420 DEBUG:boto:Method: GET 2023-09-29T22:17:25.420 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.420 DEBUG:boto:Data: 2023-09-29T22:17:25.420 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.421 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.421 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.421 DEBUG:boto:Params: {} 2023-09-29T22:17:25.421 DEBUG:boto:Token: None 2023-09-29T22:17:25.421 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b314fd8781d2b36-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.423 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.423 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.424 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.424 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.424 DEBUG:boto:Method: GET 2023-09-29T22:17:25.424 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.424 DEBUG:boto:Data: 2023-09-29T22:17:25.424 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.424 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.424 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.424 DEBUG:boto:Params: {} 2023-09-29T22:17:25.424 DEBUG:boto:Token: None 2023-09-29T22:17:25.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008098d0f2df1acd0b-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.427 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.427 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.427 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.427 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.428 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.428 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.428 DEBUG:boto:Data: 2023-09-29T22:17:25.428 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.428 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.428 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.428 DEBUG:boto:Params: {} 2023-09-29T22:17:25.428 DEBUG:boto:Token: None 2023-09-29T22:17:25.428 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E8b+WEv/Bbl2llbHxWAuhZDhHL0= 2023-09-29T22:17:25.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E8b+WEv/Bbl2llbHxWAuhZDhHL0=', 'Content-Length': '0'} 2023-09-29T22:17:25.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d206fd7849d00e9-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.430 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.431 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.431 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.431 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.431 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.431 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.431 DEBUG:boto:Data: 2023-09-29T22:17:25.431 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.431 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.431 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.431 DEBUG:boto:Params: {} 2023-09-29T22:17:25.432 DEBUG:boto:Token: None 2023-09-29T22:17:25.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.432 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E8b+WEv/Bbl2llbHxWAuhZDhHL0= 2023-09-29T22:17:25.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E8b+WEv/Bbl2llbHxWAuhZDhHL0=', 'Content-Length': '0'} 2023-09-29T22:17:25.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056f5df50f45a6035-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:25.434 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.434 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.434 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.434 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.435 DEBUG:boto:Method: GET 2023-09-29T22:17:25.435 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.435 DEBUG:boto:Data: 2023-09-29T22:17:25.435 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.435 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.435 DEBUG:boto:Params: {} 2023-09-29T22:17:25.435 DEBUG:boto:Token: None 2023-09-29T22:17:25.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.436 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001600a79731f35083-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.438 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.438 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.438 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.438 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.438 DEBUG:boto:Method: GET 2023-09-29T22:17:25.438 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.438 DEBUG:boto:Data: 2023-09-29T22:17:25.438 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.438 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.439 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.439 DEBUG:boto:Params: {} 2023-09-29T22:17:25.439 DEBUG:boto:Token: None 2023-09-29T22:17:25.439 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.439 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be50b09dcdb6b813-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.482 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.483 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.483 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.483 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.483 DEBUG:boto:Method: GET 2023-09-29T22:17:25.483 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.483 DEBUG:boto:Data: 2023-09-29T22:17:25.483 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.484 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.484 DEBUG:boto:Params: {} 2023-09-29T22:17:25.484 DEBUG:boto:Token: None 2023-09-29T22:17:25.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504eccd58999cf2f-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.487 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.487 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.487 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.488 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.488 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.488 DEBUG:boto:Method: GET 2023-09-29T22:17:25.488 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.488 DEBUG:boto:Data: 2023-09-29T22:17:25.488 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.488 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.488 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.489 DEBUG:boto:Params: {} 2023-09-29T22:17:25.489 DEBUG:boto:Token: None 2023-09-29T22:17:25.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.489 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ca82938f5d9b3f-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.534 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.535 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.535 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.535 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.535 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.535 DEBUG:boto:Data: 2023-09-29T22:17:25.535 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.536 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.536 DEBUG:boto:Params: {} 2023-09-29T22:17:25.536 DEBUG:boto:Token: None 2023-09-29T22:17:25.536 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:slZpJcKjFDqL1rc7hrsftPmF0dY= 2023-09-29T22:17:25.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:slZpJcKjFDqL1rc7hrsftPmF0dY=', 'Content-Length': '0'} 2023-09-29T22:17:25.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a1bcb9365370838-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.539 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.539 DEBUG:boto:Method: HEAD 2023-09-29T22:17:25.539 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.539 DEBUG:boto:Data: 2023-09-29T22:17:25.539 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.539 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.539 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.540 DEBUG:boto:Params: {} 2023-09-29T22:17:25.540 DEBUG:boto:Token: None 2023-09-29T22:17:25.540 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:slZpJcKjFDqL1rc7hrsftPmF0dY= 2023-09-29T22:17:25.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:slZpJcKjFDqL1rc7hrsftPmF0dY=', 'Content-Length': '0'} 2023-09-29T22:17:25.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000157e4f6cd9281434-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:25.543 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.543 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.543 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.543 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.543 DEBUG:boto:Method: GET 2023-09-29T22:17:25.543 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.543 DEBUG:boto:Data: 2023-09-29T22:17:25.543 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.543 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.543 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.544 DEBUG:boto:Params: {} 2023-09-29T22:17:25.544 DEBUG:boto:Token: None 2023-09-29T22:17:25.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fadaaa82689075b-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.586 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.587 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:25.587 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.587 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.587 DEBUG:boto:Method: GET 2023-09-29T22:17:25.587 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.587 DEBUG:boto:Data: 2023-09-29T22:17:25.587 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.587 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.587 DEBUG:boto:Params: {} 2023-09-29T22:17:25.588 DEBUG:boto:Token: None 2023-09-29T22:17:25.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:17:25.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU= 2023-09-29T22:17:25.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3QNaAxR2j8TKMVezzGAHmxhEHYU=', 'Content-Length': '0'} 2023-09-29T22:17:25.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029ce720a3a08c311-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.634 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.635 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.635 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.635 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.635 DEBUG:boto:Method: GET 2023-09-29T22:17:25.635 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.635 DEBUG:boto:Data: 2023-09-29T22:17:25.635 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.635 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:25.635 DEBUG:boto:Port: 8001 2023-09-29T22:17:25.636 DEBUG:boto:Params: {} 2023-09-29T22:17:25.636 DEBUG:boto:Token: None 2023-09-29T22:17:25.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5407a3bc349511-0065174cf5-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.639 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.639 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:25.639 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:25.639 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.640 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.640 DEBUG:boto:Method: GET 2023-09-29T22:17:25.640 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:17:25.640 DEBUG:boto:Data: 2023-09-29T22:17:25.640 DEBUG:boto:Headers: {} 2023-09-29T22:17:25.640 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:25.640 DEBUG:boto:Port: 8000 2023-09-29T22:17:25.640 DEBUG:boto:Params: {} 2023-09-29T22:17:25.640 DEBUG:boto:Token: None 2023-09-29T22:17:25.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:25 GMT /dmuhsu-16/ 2023-09-29T22:17:25.641 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g= 2023-09-29T22:17:25.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eOFYHnO13jIBQNX+o0coS4r6S9g=', 'Content-Length': '0'} 2023-09-29T22:17:25.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7c4f90f6a6f754-0065174cf5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:25.643 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:25.644 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone3, test-zone2} 2023-09-29T22:17:25.644 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-17 2023-09-29T22:17:25.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.644 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:25.703 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:25.704 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:25.742 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:25.742+0000 7fa46f80f880 1 bucket sync caught up with source: 2023-09-29T22:17:25.742 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.22.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:17:25.742 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.22.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:17:25.742 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:25.742+0000 7fa46f80f880 0 bucket checkpoint complete 2023-09-29T22:17:25.755 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-17 2023-09-29T22:17:25.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.755 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:25.813 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:25.814 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:25.852+0000 7efdcbf36880 1 bucket sync caught up with source: 2023-09-29T22:17:25.852 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.22.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:17:25.852 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.22.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:17:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:25.852+0000 7efdcbf36880 0 bucket checkpoint complete 2023-09-29T22:17:25.866 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-17 2023-09-29T22:17:25.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.867 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:25.919 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:25.919 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:25.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:25.955+0000 7fbc2f327880 1 bucket sync caught up with source: 2023-09-29T22:17:25.955 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.27.10, , 00000000010.36.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:17:25.955 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.27.10, , 00000000010.36.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:17:25.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:25.955+0000 7fbc2f327880 0 bucket checkpoint complete 2023-09-29T22:17:25.967 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-17 2023-09-29T22:17:25.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:25.967 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:26.018 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:26.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:26.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:26.052+0000 7f7691936880 1 bucket sync caught up with source: 2023-09-29T22:17:26.052 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.27.10, , 00000000010.36.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:17:26.052 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.27.10, , 00000000010.36.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:17:26.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:26.052+0000 7f7691936880 0 bucket checkpoint complete 2023-09-29T22:17:26.066 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-17 2023-09-29T22:17:26.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:26.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:26.067 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:26.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:26.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:26.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:26.153+0000 7f937200f880 1 bucket sync caught up with source: 2023-09-29T22:17:26.153 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000003.21.10, , 00000000014.49.10, , , , , ] 2023-09-29T22:17:26.153 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000003.21.10, , 00000000014.49.10, , , , , ] 2023-09-29T22:17:26.153 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:26.153+0000 7f937200f880 0 bucket checkpoint complete 2023-09-29T22:17:26.165 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-17 2023-09-29T22:17:26.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:26.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:26.165 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:26.224 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:26.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:26.263 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:26.263+0000 7fec6fda2880 1 bucket sync caught up with source: 2023-09-29T22:17:26.263 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.29.10, , 00000000003.21.10, , 00000000014.49.10, , , , , ] 2023-09-29T22:17:26.263 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.29.10, , 00000000003.21.10, , 00000000014.49.10, , , , , ] 2023-09-29T22:17:26.263 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:26.263+0000 7fec6fda2880 0 bucket checkpoint complete 2023-09-29T22:17:26.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone2, test-zone1} 2023-09-29T22:17:26.276 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.276 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.276 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.277 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:26.277 DEBUG:boto:Data: 2023-09-29T22:17:26.277 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.277 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.277 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.277 DEBUG:boto:Params: {} 2023-09-29T22:17:26.277 DEBUG:boto:Token: None 2023-09-29T22:17:26.277 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.278 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1cWRPdG/0nZJSm2aPkiaJDoQzTQ= 2023-09-29T22:17:26.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1cWRPdG/0nZJSm2aPkiaJDoQzTQ=', 'Content-Length': '0'} 2023-09-29T22:17:26.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c544d1e33d4f2f7-0065174cf6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.280 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.280 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.280 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.280 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:26.280 DEBUG:boto:Data: 2023-09-29T22:17:26.280 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.281 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.281 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.281 DEBUG:boto:Params: {} 2023-09-29T22:17:26.281 DEBUG:boto:Token: None 2023-09-29T22:17:26.281 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.281 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1cWRPdG/0nZJSm2aPkiaJDoQzTQ= 2023-09-29T22:17:26.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1cWRPdG/0nZJSm2aPkiaJDoQzTQ=', 'Content-Length': '0'} 2023-09-29T22:17:26.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f6ecc7ae8aac081-0065174cf6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:26.284 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.284 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.284 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.284 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.284 DEBUG:boto:Method: GET 2023-09-29T22:17:26.284 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.284 DEBUG:boto:Data: 2023-09-29T22:17:26.284 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.284 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.284 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.285 DEBUG:boto:Params: {} 2023-09-29T22:17:26.285 DEBUG:boto:Token: None 2023-09-29T22:17:26.285 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/?versions 2023-09-29T22:17:26.285 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw= 2023-09-29T22:17:26.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw=', 'Content-Length': '0'} 2023-09-29T22:17:26.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5223db5017cd004-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.287 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:26.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:26.288 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.289 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.289 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.289 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.289 DEBUG:boto:Method: GET 2023-09-29T22:17:26.289 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.289 DEBUG:boto:Data: 2023-09-29T22:17:26.289 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.289 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.289 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.289 DEBUG:boto:Params: {} 2023-09-29T22:17:26.290 DEBUG:boto:Token: None 2023-09-29T22:17:26.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/?versions 2023-09-29T22:17:26.290 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw= 2023-09-29T22:17:26.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw=', 'Content-Length': '0'} 2023-09-29T22:17:26.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf67abab80d3c74e-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.292 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.293 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:26.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:26.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:26.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:26.293 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.294 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.294 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.294 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.294 DEBUG:boto:Method: GET 2023-09-29T22:17:26.294 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.294 DEBUG:boto:Data: 2023-09-29T22:17:26.294 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.294 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.294 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.294 DEBUG:boto:Params: {} 2023-09-29T22:17:26.295 DEBUG:boto:Token: None 2023-09-29T22:17:26.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/?versions 2023-09-29T22:17:26.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw= 2023-09-29T22:17:26.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw=', 'Content-Length': '0'} 2023-09-29T22:17:26.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0873e1e1be755f-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.297 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.298 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.298 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.298 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.298 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.298 DEBUG:boto:Method: GET 2023-09-29T22:17:26.299 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:26.299 DEBUG:boto:Data: 2023-09-29T22:17:26.299 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.299 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.299 DEBUG:boto:Params: {} 2023-09-29T22:17:26.299 DEBUG:boto:Token: None 2023-09-29T22:17:26.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/?versions 2023-09-29T22:17:26.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw= 2023-09-29T22:17:26.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVXeaqjlZwSyH5suPSmG7gU+jcw=', 'Content-Length': '0'} 2023-09-29T22:17:26.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5c94de92f39164-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.302 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:26.302 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.302 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.303 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.303 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.303 DEBUG:boto:Method: GET 2023-09-29T22:17:26.303 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.303 DEBUG:boto:Data: 2023-09-29T22:17:26.303 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.303 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.303 DEBUG:boto:Params: {} 2023-09-29T22:17:26.304 DEBUG:boto:Token: None 2023-09-29T22:17:26.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b7d670b9dfc51e9-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.306 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.306 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.306 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.306 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.306 DEBUG:boto:Method: GET 2023-09-29T22:17:26.306 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.306 DEBUG:boto:Data: 2023-09-29T22:17:26.306 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.306 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.306 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.307 DEBUG:boto:Params: {} 2023-09-29T22:17:26.307 DEBUG:boto:Token: None 2023-09-29T22:17:26.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1357bd077d4f343-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.309 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.309 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.309 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.309 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.309 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.310 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.310 DEBUG:boto:Data: 2023-09-29T22:17:26.310 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.310 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.310 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.310 DEBUG:boto:Params: {} 2023-09-29T22:17:26.310 DEBUG:boto:Token: None 2023-09-29T22:17:26.310 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.310 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KNoAShacMFrRobJ1B2Ygd7ljv1k= 2023-09-29T22:17:26.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KNoAShacMFrRobJ1B2Ygd7ljv1k=', 'Content-Length': '0'} 2023-09-29T22:17:26.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009859589dc22edf7c-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.312 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.312 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.312 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.312 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.313 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.313 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.313 DEBUG:boto:Data: 2023-09-29T22:17:26.313 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.313 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.313 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.313 DEBUG:boto:Params: {} 2023-09-29T22:17:26.313 DEBUG:boto:Token: None 2023-09-29T22:17:26.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.314 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KNoAShacMFrRobJ1B2Ygd7ljv1k= 2023-09-29T22:17:26.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KNoAShacMFrRobJ1B2Ygd7ljv1k=', 'Content-Length': '0'} 2023-09-29T22:17:26.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034671cb4697f62b9-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:26.315 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.316 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.316 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.316 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.316 DEBUG:boto:Method: GET 2023-09-29T22:17:26.316 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.316 DEBUG:boto:Data: 2023-09-29T22:17:26.316 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.316 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.316 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.317 DEBUG:boto:Params: {} 2023-09-29T22:17:26.317 DEBUG:boto:Token: None 2023-09-29T22:17:26.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.317 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019e08195dd35938d-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.319 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.319 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.319 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.319 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.319 DEBUG:boto:Method: GET 2023-09-29T22:17:26.319 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.319 DEBUG:boto:Data: 2023-09-29T22:17:26.319 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.320 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.320 DEBUG:boto:Params: {} 2023-09-29T22:17:26.320 DEBUG:boto:Token: None 2023-09-29T22:17:26.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c28d079667abdbe-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.322 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.322 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.322 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.322 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.322 DEBUG:boto:Method: GET 2023-09-29T22:17:26.322 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.322 DEBUG:boto:Data: 2023-09-29T22:17:26.323 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.323 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.323 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.323 DEBUG:boto:Params: {} 2023-09-29T22:17:26.323 DEBUG:boto:Token: None 2023-09-29T22:17:26.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017016ca46103783-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.370 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.371 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.371 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.371 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.371 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.371 DEBUG:boto:Method: GET 2023-09-29T22:17:26.371 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.372 DEBUG:boto:Data: 2023-09-29T22:17:26.372 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.372 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.372 DEBUG:boto:Params: {} 2023-09-29T22:17:26.372 DEBUG:boto:Token: None 2023-09-29T22:17:26.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec393b5391eaa513-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.375 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.375 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.375 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.375 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.375 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:26.375 DEBUG:boto:Data: 2023-09-29T22:17:26.376 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.376 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.376 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.376 DEBUG:boto:Params: {} 2023-09-29T22:17:26.376 DEBUG:boto:Token: None 2023-09-29T22:17:26.376 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy 2023-09-29T22:17:26.376 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KdPyQepTbJ6sLwF43Jm4Q320Fw4= 2023-09-29T22:17:26.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KdPyQepTbJ6sLwF43Jm4Q320Fw4=', 'Content-Length': '0'} 2023-09-29T22:17:26.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d18f01d5414a2fa8-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.378 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.379 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.379 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.379 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:26.379 DEBUG:boto:Data: 2023-09-29T22:17:26.379 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.379 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.379 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.379 DEBUG:boto:Params: {} 2023-09-29T22:17:26.380 DEBUG:boto:Token: None 2023-09-29T22:17:26.380 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy 2023-09-29T22:17:26.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KdPyQepTbJ6sLwF43Jm4Q320Fw4= 2023-09-29T22:17:26.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KdPyQepTbJ6sLwF43Jm4Q320Fw4=', 'Content-Length': '0'} 2023-09-29T22:17:26.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024cb465b27c39057-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:26.382 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.382 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.382 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.382 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.382 DEBUG:boto:Method: GET 2023-09-29T22:17:26.383 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.383 DEBUG:boto:Data: 2023-09-29T22:17:26.383 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.383 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.383 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.383 DEBUG:boto:Params: {} 2023-09-29T22:17:26.383 DEBUG:boto:Token: None 2023-09-29T22:17:26.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049f808b44e27f27f-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.386 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:26.386 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:26.386 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.386 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.386 DEBUG:boto:Method: GET 2023-09-29T22:17:26.386 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.386 DEBUG:boto:Data: 2023-09-29T22:17:26.386 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.386 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.386 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.387 DEBUG:boto:Params: {} 2023-09-29T22:17:26.387 DEBUG:boto:Token: None 2023-09-29T22:17:26.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:26.387 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk= 2023-09-29T22:17:26.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XWy1uVDYzUXoG7PNsd6endLonVk=', 'Content-Length': '0'} 2023-09-29T22:17:26.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a76b10dbae95eadd-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.389 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.389 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.389 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.389 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.389 DEBUG:boto:Method: GET 2023-09-29T22:17:26.389 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.390 DEBUG:boto:Data: 2023-09-29T22:17:26.390 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.390 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.390 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.390 DEBUG:boto:Params: {} 2023-09-29T22:17:26.390 DEBUG:boto:Token: None 2023-09-29T22:17:26.390 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.390 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024d1be6d7913bd37-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.392 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.393 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.393 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.393 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.393 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.393 DEBUG:boto:Method: GET 2023-09-29T22:17:26.393 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:26.393 DEBUG:boto:Data: 2023-09-29T22:17:26.394 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.394 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.394 DEBUG:boto:Params: {} 2023-09-29T22:17:26.394 DEBUG:boto:Token: None 2023-09-29T22:17:26.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.394 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002881b0b912ec6d52-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.396 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:26.397 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.397 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.397 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.397 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.397 DEBUG:boto:Method: GET 2023-09-29T22:17:26.398 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.398 DEBUG:boto:Data: 2023-09-29T22:17:26.398 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.398 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.398 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.398 DEBUG:boto:Params: {} 2023-09-29T22:17:26.398 DEBUG:boto:Token: None 2023-09-29T22:17:26.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.398 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001d6d6f278183b7c-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.400 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.400 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.401 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.401 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.401 DEBUG:boto:Method: GET 2023-09-29T22:17:26.401 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.401 DEBUG:boto:Data: 2023-09-29T22:17:26.401 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.401 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.401 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.401 DEBUG:boto:Params: {} 2023-09-29T22:17:26.402 DEBUG:boto:Token: None 2023-09-29T22:17:26.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000410e57a3a4ff4916-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.404 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.404 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.404 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.404 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.404 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.404 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.404 DEBUG:boto:Data: 2023-09-29T22:17:26.404 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.405 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.405 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.405 DEBUG:boto:Params: {} 2023-09-29T22:17:26.405 DEBUG:boto:Token: None 2023-09-29T22:17:26.405 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.405 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TuUUfYjY7jM8n3A4aRfcxn0QoX8= 2023-09-29T22:17:26.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TuUUfYjY7jM8n3A4aRfcxn0QoX8=', 'Content-Length': '0'} 2023-09-29T22:17:26.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae66bedd7d61449b-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.407 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.407 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.407 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.407 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.407 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.407 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.408 DEBUG:boto:Data: 2023-09-29T22:17:26.408 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.408 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.408 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.408 DEBUG:boto:Params: {} 2023-09-29T22:17:26.408 DEBUG:boto:Token: None 2023-09-29T22:17:26.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.408 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TuUUfYjY7jM8n3A4aRfcxn0QoX8= 2023-09-29T22:17:26.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TuUUfYjY7jM8n3A4aRfcxn0QoX8=', 'Content-Length': '0'} 2023-09-29T22:17:26.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0e4e10fa17ff89-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:26.410 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.410 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.410 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.411 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.411 DEBUG:boto:Method: GET 2023-09-29T22:17:26.411 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.411 DEBUG:boto:Data: 2023-09-29T22:17:26.411 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.411 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.411 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.411 DEBUG:boto:Params: {} 2023-09-29T22:17:26.411 DEBUG:boto:Token: None 2023-09-29T22:17:26.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9278ef22397e2a6-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.454 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.455 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.455 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.455 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.455 DEBUG:boto:Method: GET 2023-09-29T22:17:26.455 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.455 DEBUG:boto:Data: 2023-09-29T22:17:26.455 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.455 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.455 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.455 DEBUG:boto:Params: {} 2023-09-29T22:17:26.456 DEBUG:boto:Token: None 2023-09-29T22:17:26.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040f7771c697d0b53-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.458 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.458 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.459 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.459 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.459 DEBUG:boto:Method: GET 2023-09-29T22:17:26.459 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.459 DEBUG:boto:Data: 2023-09-29T22:17:26.459 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.459 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.459 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.459 DEBUG:boto:Params: {} 2023-09-29T22:17:26.459 DEBUG:boto:Token: None 2023-09-29T22:17:26.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000727d16e59ae1cc84-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.502 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.503 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.503 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.503 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.503 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.503 DEBUG:boto:Method: GET 2023-09-29T22:17:26.504 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.504 DEBUG:boto:Data: 2023-09-29T22:17:26.504 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.504 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.504 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.504 DEBUG:boto:Params: {} 2023-09-29T22:17:26.504 DEBUG:boto:Token: None 2023-09-29T22:17:26.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002558ce993d64acd4-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.507 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.507 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.507 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.507 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.508 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.508 DEBUG:boto:Data: 2023-09-29T22:17:26.508 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.508 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.508 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.508 DEBUG:boto:Params: {} 2023-09-29T22:17:26.508 DEBUG:boto:Token: None 2023-09-29T22:17:26.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u36sY58xR8/jo4Xtf6DDieVfZhI= 2023-09-29T22:17:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u36sY58xR8/jo4Xtf6DDieVfZhI=', 'Content-Length': '0'} 2023-09-29T22:17:26.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006da0c394f48f80c3-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.511 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.511 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.511 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.511 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.511 DEBUG:boto:Data: 2023-09-29T22:17:26.511 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.511 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.511 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.511 DEBUG:boto:Params: {} 2023-09-29T22:17:26.512 DEBUG:boto:Token: None 2023-09-29T22:17:26.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u36sY58xR8/jo4Xtf6DDieVfZhI= 2023-09-29T22:17:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u36sY58xR8/jo4Xtf6DDieVfZhI=', 'Content-Length': '0'} 2023-09-29T22:17:26.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000350b356b1193eba2-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:26.514 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.514 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.514 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.514 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.514 DEBUG:boto:Method: GET 2023-09-29T22:17:26.515 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.515 DEBUG:boto:Data: 2023-09-29T22:17:26.515 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.515 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.515 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.515 DEBUG:boto:Params: {} 2023-09-29T22:17:26.515 DEBUG:boto:Token: None 2023-09-29T22:17:26.515 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.515 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003acf36a588dd3bf5-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.558 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.559 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:26.559 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.559 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.559 DEBUG:boto:Method: GET 2023-09-29T22:17:26.559 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.559 DEBUG:boto:Data: 2023-09-29T22:17:26.559 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.559 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.559 DEBUG:boto:Params: {} 2023-09-29T22:17:26.560 DEBUG:boto:Token: None 2023-09-29T22:17:26.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:26.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU= 2023-09-29T22:17:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tODj08pKvsRoCt69RDaYCK61/dU=', 'Content-Length': '0'} 2023-09-29T22:17:26.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039ab7a98eb4e79ad-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.562 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.562 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.562 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.562 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.563 DEBUG:boto:Method: GET 2023-09-29T22:17:26.563 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.563 DEBUG:boto:Data: 2023-09-29T22:17:26.563 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.563 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.563 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.563 DEBUG:boto:Params: {} 2023-09-29T22:17:26.563 DEBUG:boto:Token: None 2023-09-29T22:17:26.563 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352212ab4b67e1df-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.606 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.607 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.607 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.607 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.607 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.607 DEBUG:boto:Method: GET 2023-09-29T22:17:26.608 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:26.608 DEBUG:boto:Data: 2023-09-29T22:17:26.608 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.608 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.608 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.608 DEBUG:boto:Params: {} 2023-09-29T22:17:26.608 DEBUG:boto:Token: None 2023-09-29T22:17:26.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.608 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8a4d45a587a4c2-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.611 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:26.612 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.612 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.612 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.612 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.612 DEBUG:boto:Method: GET 2023-09-29T22:17:26.612 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.612 DEBUG:boto:Data: 2023-09-29T22:17:26.612 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.612 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.613 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.613 DEBUG:boto:Params: {} 2023-09-29T22:17:26.613 DEBUG:boto:Token: None 2023-09-29T22:17:26.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006264a20ff0067425-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.615 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.615 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.615 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.615 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.616 DEBUG:boto:Method: GET 2023-09-29T22:17:26.616 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.616 DEBUG:boto:Data: 2023-09-29T22:17:26.616 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.616 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.616 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.616 DEBUG:boto:Params: {} 2023-09-29T22:17:26.616 DEBUG:boto:Token: None 2023-09-29T22:17:26.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.617 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089da4fdd8ffbc778-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.658 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.659 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.659 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.659 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.659 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.659 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.659 DEBUG:boto:Data: 2023-09-29T22:17:26.659 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.659 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.659 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.660 DEBUG:boto:Params: {} 2023-09-29T22:17:26.660 DEBUG:boto:Token: None 2023-09-29T22:17:26.660 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6VRRTkFsBRWtIod2CHRnQakguuY= 2023-09-29T22:17:26.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6VRRTkFsBRWtIod2CHRnQakguuY=', 'Content-Length': '0'} 2023-09-29T22:17:26.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038d1d28a4b64791d-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.662 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.662 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.663 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.663 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.663 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.663 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.663 DEBUG:boto:Data: 2023-09-29T22:17:26.663 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.663 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.663 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.663 DEBUG:boto:Params: {} 2023-09-29T22:17:26.663 DEBUG:boto:Token: None 2023-09-29T22:17:26.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6VRRTkFsBRWtIod2CHRnQakguuY= 2023-09-29T22:17:26.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6VRRTkFsBRWtIod2CHRnQakguuY=', 'Content-Length': '0'} 2023-09-29T22:17:26.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006942f1e7af2517a1-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:26.666 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.666 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.666 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.666 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.666 DEBUG:boto:Method: GET 2023-09-29T22:17:26.667 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.667 DEBUG:boto:Data: 2023-09-29T22:17:26.667 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.667 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.667 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.667 DEBUG:boto:Params: {} 2023-09-29T22:17:26.667 DEBUG:boto:Token: None 2023-09-29T22:17:26.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.667 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee1c8479ec0c7fa4-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.669 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.669 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.670 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.670 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.670 DEBUG:boto:Method: GET 2023-09-29T22:17:26.670 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.670 DEBUG:boto:Data: 2023-09-29T22:17:26.670 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.670 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.670 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.670 DEBUG:boto:Params: {} 2023-09-29T22:17:26.670 DEBUG:boto:Token: None 2023-09-29T22:17:26.671 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.671 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bce5c76daf7bbebf-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.714 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.715 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.715 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.715 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.715 DEBUG:boto:Method: GET 2023-09-29T22:17:26.715 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.715 DEBUG:boto:Data: 2023-09-29T22:17:26.715 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.715 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.716 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.716 DEBUG:boto:Params: {} 2023-09-29T22:17:26.716 DEBUG:boto:Token: None 2023-09-29T22:17:26.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0acd064e0d69fe-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.719 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.719 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.719 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.720 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.720 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.720 DEBUG:boto:Method: GET 2023-09-29T22:17:26.720 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.720 DEBUG:boto:Data: 2023-09-29T22:17:26.720 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.720 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.721 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.721 DEBUG:boto:Params: {} 2023-09-29T22:17:26.721 DEBUG:boto:Token: None 2023-09-29T22:17:26.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.721 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ebbdf3c858b1fcf-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.770 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.771 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.771 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.771 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.771 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.771 DEBUG:boto:Data: 2023-09-29T22:17:26.771 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.771 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.772 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.772 DEBUG:boto:Params: {} 2023-09-29T22:17:26.772 DEBUG:boto:Token: None 2023-09-29T22:17:26.772 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jMwYbSUFPghgJx/4t3sPO5N3TyY= 2023-09-29T22:17:26.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jMwYbSUFPghgJx/4t3sPO5N3TyY=', 'Content-Length': '0'} 2023-09-29T22:17:26.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d43c8233d021a938-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.774 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.775 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.775 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.775 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.775 DEBUG:boto:Data: 2023-09-29T22:17:26.775 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.775 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.775 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.775 DEBUG:boto:Params: {} 2023-09-29T22:17:26.775 DEBUG:boto:Token: None 2023-09-29T22:17:26.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.776 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jMwYbSUFPghgJx/4t3sPO5N3TyY= 2023-09-29T22:17:26.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jMwYbSUFPghgJx/4t3sPO5N3TyY=', 'Content-Length': '0'} 2023-09-29T22:17:26.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033fc8b4c4249223c-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:26.778 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.778 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.778 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.779 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.779 DEBUG:boto:Method: GET 2023-09-29T22:17:26.779 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.779 DEBUG:boto:Data: 2023-09-29T22:17:26.779 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.779 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.779 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.779 DEBUG:boto:Params: {} 2023-09-29T22:17:26.780 DEBUG:boto:Token: None 2023-09-29T22:17:26.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5488bbc067c5f7b-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.782 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.782 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:26.782 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.782 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.782 DEBUG:boto:Method: GET 2023-09-29T22:17:26.782 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.782 DEBUG:boto:Data: 2023-09-29T22:17:26.783 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.783 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.783 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.783 DEBUG:boto:Params: {} 2023-09-29T22:17:26.783 DEBUG:boto:Token: None 2023-09-29T22:17:26.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:26.783 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok= 2023-09-29T22:17:26.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kXpE72Otclmff8rtqn1E9ldIVok=', 'Content-Length': '0'} 2023-09-29T22:17:26.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb1d116096573e9-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.826 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.826 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.827 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.827 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.827 DEBUG:boto:Method: GET 2023-09-29T22:17:26.827 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.827 DEBUG:boto:Data: 2023-09-29T22:17:26.827 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.827 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.827 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.827 DEBUG:boto:Params: {} 2023-09-29T22:17:26.827 DEBUG:boto:Token: None 2023-09-29T22:17:26.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2d4ccc1b64d8cc4-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.830 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.830 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.831 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.831 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.831 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.831 DEBUG:boto:Method: GET 2023-09-29T22:17:26.831 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:26.831 DEBUG:boto:Data: 2023-09-29T22:17:26.831 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.831 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.831 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.831 DEBUG:boto:Params: {} 2023-09-29T22:17:26.832 DEBUG:boto:Token: None 2023-09-29T22:17:26.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093012be897d10d09-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.834 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:26.835 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.835 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.835 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.835 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.835 DEBUG:boto:Method: GET 2023-09-29T22:17:26.835 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.835 DEBUG:boto:Data: 2023-09-29T22:17:26.835 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.835 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.835 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.836 DEBUG:boto:Params: {} 2023-09-29T22:17:26.836 DEBUG:boto:Token: None 2023-09-29T22:17:26.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs= 2023-09-29T22:17:26.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs=', 'Content-Length': '0'} 2023-09-29T22:17:26.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ff376ead23c09f2-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.838 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.838 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.838 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.838 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.838 DEBUG:boto:Method: GET 2023-09-29T22:17:26.838 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.839 DEBUG:boto:Data: 2023-09-29T22:17:26.839 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.839 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.839 DEBUG:boto:Params: {} 2023-09-29T22:17:26.839 DEBUG:boto:Token: None 2023-09-29T22:17:26.839 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.839 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs= 2023-09-29T22:17:26.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs=', 'Content-Length': '0'} 2023-09-29T22:17:26.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090425a8e2787368f-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.841 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.841 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.841 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.841 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.841 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.842 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.842 DEBUG:boto:Data: 2023-09-29T22:17:26.842 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.842 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.842 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.842 DEBUG:boto:Params: {} 2023-09-29T22:17:26.842 DEBUG:boto:Token: None 2023-09-29T22:17:26.842 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F5gJwxcjX4qsmYqH5uxZLXZFlA4= 2023-09-29T22:17:26.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F5gJwxcjX4qsmYqH5uxZLXZFlA4=', 'Content-Length': '0'} 2023-09-29T22:17:26.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022815580a8241f55-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.844 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.844 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.844 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.845 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.845 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.845 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.845 DEBUG:boto:Data: 2023-09-29T22:17:26.845 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.845 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.845 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.845 DEBUG:boto:Params: {} 2023-09-29T22:17:26.845 DEBUG:boto:Token: None 2023-09-29T22:17:26.845 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F5gJwxcjX4qsmYqH5uxZLXZFlA4= 2023-09-29T22:17:26.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F5gJwxcjX4qsmYqH5uxZLXZFlA4=', 'Content-Length': '0'} 2023-09-29T22:17:26.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4915d29b4b4849c-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:26.848 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.848 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.848 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.848 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.848 DEBUG:boto:Method: GET 2023-09-29T22:17:26.848 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.848 DEBUG:boto:Data: 2023-09-29T22:17:26.848 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.848 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.848 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.849 DEBUG:boto:Params: {} 2023-09-29T22:17:26.849 DEBUG:boto:Token: None 2023-09-29T22:17:26.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.849 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs= 2023-09-29T22:17:26.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs=', 'Content-Length': '0'} 2023-09-29T22:17:26.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a372babef4c5251-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.894 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.895 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.895 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.895 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.895 DEBUG:boto:Method: GET 2023-09-29T22:17:26.895 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.895 DEBUG:boto:Data: 2023-09-29T22:17:26.895 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.897 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.897 DEBUG:boto:Params: {} 2023-09-29T22:17:26.897 DEBUG:boto:Token: None 2023-09-29T22:17:26.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:26.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs= 2023-09-29T22:17:26.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2rYk9PmRooIJbXjcSSihNdQaHXs=', 'Content-Length': '0'} 2023-09-29T22:17:26.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c27defccaa3b3ba-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.942 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.943 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.943 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.943 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.943 DEBUG:boto:Method: GET 2023-09-29T22:17:26.943 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.943 DEBUG:boto:Data: 2023-09-29T22:17:26.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.944 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.944 DEBUG:boto:Params: {} 2023-09-29T22:17:26.944 DEBUG:boto:Token: None 2023-09-29T22:17:26.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d17e887f4e701abb-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.947 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.948 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:26.948 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:26.948 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.948 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.948 DEBUG:boto:Method: GET 2023-09-29T22:17:26.948 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:26.948 DEBUG:boto:Data: 2023-09-29T22:17:26.948 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:26.948 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.949 DEBUG:boto:Params: {} 2023-09-29T22:17:26.949 DEBUG:boto:Token: None 2023-09-29T22:17:26.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/ 2023-09-29T22:17:26.950 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930= 2023-09-29T22:17:26.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DWxOcG6/aee+zc/MnaVpZB6w930=', 'Content-Length': '0'} 2023-09-29T22:17:26.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002916ac6cfaca4bf3-0065174cf6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.994 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:26.995 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.995 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.995 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.996 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.996 DEBUG:boto:Data: 2023-09-29T22:17:26.996 DEBUG:boto:Headers: {} 2023-09-29T22:17:26.996 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:26.996 DEBUG:boto:Port: 8000 2023-09-29T22:17:26.996 DEBUG:boto:Params: {} 2023-09-29T22:17:26.996 DEBUG:boto:Token: None 2023-09-29T22:17:26.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:26 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I2UsjTyfujKO1sAVeBbsqza55bc= 2023-09-29T22:17:26.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I2UsjTyfujKO1sAVeBbsqza55bc=', 'Content-Length': '0'} 2023-09-29T22:17:26.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d205014f45217765-0065174cf6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:26.999 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.999 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:26.999 DEBUG:boto:Method: HEAD 2023-09-29T22:17:26.999 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.000 DEBUG:boto:Data: 2023-09-29T22:17:27.000 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.000 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.000 DEBUG:boto:Params: {} 2023-09-29T22:17:27.000 DEBUG:boto:Token: None 2023-09-29T22:17:27.000 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ= 2023-09-29T22:17:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6e7a2bf473f8099-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:27.003 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.003 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.003 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.004 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.004 DEBUG:boto:Method: GET 2023-09-29T22:17:27.004 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.004 DEBUG:boto:Data: 2023-09-29T22:17:27.004 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.005 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.005 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.005 DEBUG:boto:Params: {} 2023-09-29T22:17:27.005 DEBUG:boto:Token: None 2023-09-29T22:17:27.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b88e6a9039b589c4-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.054 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.055 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.055 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.055 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.055 DEBUG:boto:Method: GET 2023-09-29T22:17:27.055 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.055 DEBUG:boto:Data: 2023-09-29T22:17:27.055 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.055 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.055 DEBUG:boto:Params: {} 2023-09-29T22:17:27.056 DEBUG:boto:Token: None 2023-09-29T22:17:27.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce2418e80f314eff-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.101 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.101 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.102 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.102 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.102 DEBUG:boto:Method: GET 2023-09-29T22:17:27.102 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.102 DEBUG:boto:Data: 2023-09-29T22:17:27.102 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.102 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.102 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.102 DEBUG:boto:Params: {} 2023-09-29T22:17:27.102 DEBUG:boto:Token: None 2023-09-29T22:17:27.103 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.103 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a8265086378eb6-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.105 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.106 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.106 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.106 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.106 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.106 DEBUG:boto:Method: GET 2023-09-29T22:17:27.106 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.106 DEBUG:boto:Data: 2023-09-29T22:17:27.106 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.106 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.107 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.107 DEBUG:boto:Params: {} 2023-09-29T22:17:27.107 DEBUG:boto:Token: None 2023-09-29T22:17:27.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.107 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002010c337d8be78ba-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.109 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone2, test-zone1} 2023-09-29T22:17:27.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone3, test-zone1} 2023-09-29T22:17:27.110 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.110 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.110 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.110 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:27.110 DEBUG:boto:Data: 2023-09-29T22:17:27.110 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.111 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.111 DEBUG:boto:Params: {} 2023-09-29T22:17:27.111 DEBUG:boto:Token: None 2023-09-29T22:17:27.111 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY= 2023-09-29T22:17:27.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY=', 'Content-Length': '0'} 2023-09-29T22:17:27.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035cfc80ee1703bd4-0065174cf7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.113 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.113 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.113 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.114 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:27.114 DEBUG:boto:Data: 2023-09-29T22:17:27.114 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.114 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.114 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.114 DEBUG:boto:Params: {} 2023-09-29T22:17:27.114 DEBUG:boto:Token: None 2023-09-29T22:17:27.114 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.114 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY= 2023-09-29T22:17:27.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY=', 'Content-Length': '0'} 2023-09-29T22:17:27.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000011ea8980322c6bc-0065174cf7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.117 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:27.117 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.117 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.117 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.118 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.118 DEBUG:boto:Method: GET 2023-09-29T22:17:27.118 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.118 DEBUG:boto:Data: 2023-09-29T22:17:27.118 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.118 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.118 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.118 DEBUG:boto:Params: {} 2023-09-29T22:17:27.118 DEBUG:boto:Token: None 2023-09-29T22:17:27.119 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.119 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b836479e065957d0-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.121 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.122 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:27.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:27.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:27.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:27.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:27.122 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.123 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.123 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.123 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.123 DEBUG:boto:Method: GET 2023-09-29T22:17:27.123 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.123 DEBUG:boto:Data: 2023-09-29T22:17:27.123 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.123 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.123 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.123 DEBUG:boto:Params: {} 2023-09-29T22:17:27.124 DEBUG:boto:Token: None 2023-09-29T22:17:27.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001597f3897f8c1a0b-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.126 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.127 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:27.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:27.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:27.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:27.127 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.127 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.128 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.128 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.128 DEBUG:boto:Method: GET 2023-09-29T22:17:27.128 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.128 DEBUG:boto:Data: 2023-09-29T22:17:27.128 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.128 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.128 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.128 DEBUG:boto:Params: {} 2023-09-29T22:17:27.128 DEBUG:boto:Token: None 2023-09-29T22:17:27.129 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.129 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000185d46b3c624e6ac-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.131 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.132 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.132 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.132 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.132 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.132 DEBUG:boto:Method: GET 2023-09-29T22:17:27.132 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.132 DEBUG:boto:Data: 2023-09-29T22:17:27.133 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.133 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.133 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.133 DEBUG:boto:Params: {} 2023-09-29T22:17:27.133 DEBUG:boto:Token: None 2023-09-29T22:17:27.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.133 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7b26e2087f1d438-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.136 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:27.136 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.136 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.136 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.136 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.137 DEBUG:boto:Method: GET 2023-09-29T22:17:27.137 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.137 DEBUG:boto:Data: 2023-09-29T22:17:27.137 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.137 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.137 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.137 DEBUG:boto:Params: {} 2023-09-29T22:17:27.137 DEBUG:boto:Token: None 2023-09-29T22:17:27.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d41793345f27d4b7-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.140 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.140 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.140 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.140 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.140 DEBUG:boto:Method: GET 2023-09-29T22:17:27.140 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.140 DEBUG:boto:Data: 2023-09-29T22:17:27.140 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.141 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.141 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.141 DEBUG:boto:Params: {} 2023-09-29T22:17:27.141 DEBUG:boto:Token: None 2023-09-29T22:17:27.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c513e76e27318a6f-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.143 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.143 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.143 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.143 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.143 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.143 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.143 DEBUG:boto:Data: 2023-09-29T22:17:27.143 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.144 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.144 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.144 DEBUG:boto:Params: {} 2023-09-29T22:17:27.144 DEBUG:boto:Token: None 2023-09-29T22:17:27.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw= 2023-09-29T22:17:27.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw=', 'Content-Length': '0'} 2023-09-29T22:17:27.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eed0f2b2f49b87d7-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.146 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.146 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.146 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.146 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.146 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.146 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.147 DEBUG:boto:Data: 2023-09-29T22:17:27.147 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.147 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.147 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.147 DEBUG:boto:Params: {} 2023-09-29T22:17:27.147 DEBUG:boto:Token: None 2023-09-29T22:17:27.147 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.147 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw= 2023-09-29T22:17:27.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw=', 'Content-Length': '0'} 2023-09-29T22:17:27.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000546f6564691719c1-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:27.149 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.149 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.150 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.150 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.150 DEBUG:boto:Method: GET 2023-09-29T22:17:27.150 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.150 DEBUG:boto:Data: 2023-09-29T22:17:27.150 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.150 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.150 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.150 DEBUG:boto:Params: {} 2023-09-29T22:17:27.151 DEBUG:boto:Token: None 2023-09-29T22:17:27.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0ca41c35a2173aa-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.153 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.153 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.153 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.153 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.153 DEBUG:boto:Method: GET 2023-09-29T22:17:27.153 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.153 DEBUG:boto:Data: 2023-09-29T22:17:27.154 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.154 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.154 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.154 DEBUG:boto:Params: {} 2023-09-29T22:17:27.154 DEBUG:boto:Token: None 2023-09-29T22:17:27.154 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.154 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6703517879e1dcc-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.156 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.156 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.156 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.156 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.157 DEBUG:boto:Method: GET 2023-09-29T22:17:27.157 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.157 DEBUG:boto:Data: 2023-09-29T22:17:27.157 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.157 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.157 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.157 DEBUG:boto:Params: {} 2023-09-29T22:17:27.157 DEBUG:boto:Token: None 2023-09-29T22:17:27.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.158 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075fd0f1a19e26cc8-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.160 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.160 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.160 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.160 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.160 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.161 DEBUG:boto:Method: GET 2023-09-29T22:17:27.161 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.161 DEBUG:boto:Data: 2023-09-29T22:17:27.161 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.161 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.161 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.161 DEBUG:boto:Params: {} 2023-09-29T22:17:27.161 DEBUG:boto:Token: None 2023-09-29T22:17:27.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.162 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d2565f1d0360b1-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.206 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.207 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.207 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.207 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.207 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:27.207 DEBUG:boto:Data: 2023-09-29T22:17:27.208 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.208 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.208 DEBUG:boto:Params: {} 2023-09-29T22:17:27.208 DEBUG:boto:Token: None 2023-09-29T22:17:27.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy 2023-09-29T22:17:27.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l27C9iPkP9Co/ANmzwilPQFcZiw= 2023-09-29T22:17:27.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l27C9iPkP9Co/ANmzwilPQFcZiw=', 'Content-Length': '0'} 2023-09-29T22:17:27.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c84df7df4c642168-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.210 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.211 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.211 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.211 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:27.211 DEBUG:boto:Data: 2023-09-29T22:17:27.211 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.211 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.211 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.211 DEBUG:boto:Params: {} 2023-09-29T22:17:27.211 DEBUG:boto:Token: None 2023-09-29T22:17:27.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy 2023-09-29T22:17:27.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l27C9iPkP9Co/ANmzwilPQFcZiw= 2023-09-29T22:17:27.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l27C9iPkP9Co/ANmzwilPQFcZiw=', 'Content-Length': '0'} 2023-09-29T22:17:27.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6d9a12ad864b23d-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:27.214 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.214 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.214 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.214 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.214 DEBUG:boto:Method: GET 2023-09-29T22:17:27.215 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.215 DEBUG:boto:Data: 2023-09-29T22:17:27.215 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.215 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.215 DEBUG:boto:Params: {} 2023-09-29T22:17:27.215 DEBUG:boto:Token: None 2023-09-29T22:17:27.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.215 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e8122afc6156dff-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.217 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.217 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.218 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.218 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.218 DEBUG:boto:Method: GET 2023-09-29T22:17:27.218 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.218 DEBUG:boto:Data: 2023-09-29T22:17:27.218 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.218 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.218 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.218 DEBUG:boto:Params: {} 2023-09-29T22:17:27.219 DEBUG:boto:Token: None 2023-09-29T22:17:27.219 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.219 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d6b1eaff1aca75a-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.220 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.221 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.221 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.221 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.221 DEBUG:boto:Method: GET 2023-09-29T22:17:27.221 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.221 DEBUG:boto:Data: 2023-09-29T22:17:27.221 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.221 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.221 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.222 DEBUG:boto:Params: {} 2023-09-29T22:17:27.222 DEBUG:boto:Token: None 2023-09-29T22:17:27.222 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.222 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c75a71dae9e7e5-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.224 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.224 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.225 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.225 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.225 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.225 DEBUG:boto:Method: GET 2023-09-29T22:17:27.225 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:27.225 DEBUG:boto:Data: 2023-09-29T22:17:27.225 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.225 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.225 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.226 DEBUG:boto:Params: {} 2023-09-29T22:17:27.226 DEBUG:boto:Token: None 2023-09-29T22:17:27.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6335c0e333126c-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.228 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:27.229 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.229 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.229 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.229 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.229 DEBUG:boto:Method: GET 2023-09-29T22:17:27.229 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.229 DEBUG:boto:Data: 2023-09-29T22:17:27.229 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.230 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.230 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.230 DEBUG:boto:Params: {} 2023-09-29T22:17:27.230 DEBUG:boto:Token: None 2023-09-29T22:17:27.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.230 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d085998b2433e98e-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.232 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.232 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.232 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.232 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.232 DEBUG:boto:Method: GET 2023-09-29T22:17:27.233 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.233 DEBUG:boto:Data: 2023-09-29T22:17:27.233 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.233 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.233 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.233 DEBUG:boto:Params: {} 2023-09-29T22:17:27.233 DEBUG:boto:Token: None 2023-09-29T22:17:27.233 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.233 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002c3811b57946749-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.235 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.236 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.236 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.236 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.236 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.236 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.236 DEBUG:boto:Data: 2023-09-29T22:17:27.236 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.236 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.236 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.236 DEBUG:boto:Params: {} 2023-09-29T22:17:27.236 DEBUG:boto:Token: None 2023-09-29T22:17:27.237 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6heU0EziU0ifZhWj2wYeZb5a0CQ= 2023-09-29T22:17:27.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6heU0EziU0ifZhWj2wYeZb5a0CQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae3b43afb0c71f22-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.239 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.239 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.239 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.239 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.239 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.239 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.239 DEBUG:boto:Data: 2023-09-29T22:17:27.239 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.239 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.240 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.240 DEBUG:boto:Params: {} 2023-09-29T22:17:27.240 DEBUG:boto:Token: None 2023-09-29T22:17:27.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6heU0EziU0ifZhWj2wYeZb5a0CQ= 2023-09-29T22:17:27.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6heU0EziU0ifZhWj2wYeZb5a0CQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5638b101ef2cdfe-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:27.242 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.242 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.242 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.242 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.242 DEBUG:boto:Method: GET 2023-09-29T22:17:27.242 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.243 DEBUG:boto:Data: 2023-09-29T22:17:27.243 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.243 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.243 DEBUG:boto:Params: {} 2023-09-29T22:17:27.243 DEBUG:boto:Token: None 2023-09-29T22:17:27.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.243 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b915db342f8005b7-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.290 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.290 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.291 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.291 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.291 DEBUG:boto:Method: GET 2023-09-29T22:17:27.291 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.291 DEBUG:boto:Data: 2023-09-29T22:17:27.291 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.291 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.291 DEBUG:boto:Params: {} 2023-09-29T22:17:27.292 DEBUG:boto:Token: None 2023-09-29T22:17:27.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040fdbc86a0d29ba3-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.294 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.294 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.294 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.294 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.294 DEBUG:boto:Method: GET 2023-09-29T22:17:27.294 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.294 DEBUG:boto:Data: 2023-09-29T22:17:27.294 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.295 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.295 DEBUG:boto:Params: {} 2023-09-29T22:17:27.295 DEBUG:boto:Token: None 2023-09-29T22:17:27.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090be16d01ca16822-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.297 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.298 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.298 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.298 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.298 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.298 DEBUG:boto:Method: GET 2023-09-29T22:17:27.298 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.299 DEBUG:boto:Data: 2023-09-29T22:17:27.299 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.299 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.299 DEBUG:boto:Params: {} 2023-09-29T22:17:27.299 DEBUG:boto:Token: None 2023-09-29T22:17:27.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca68e61830378e8-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.302 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.302 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.302 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.302 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.302 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.303 DEBUG:boto:Data: 2023-09-29T22:17:27.303 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.303 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.303 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.303 DEBUG:boto:Params: {} 2023-09-29T22:17:27.303 DEBUG:boto:Token: None 2023-09-29T22:17:27.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kst8PAvjnErnWGNYi+wS+6pdtKs= 2023-09-29T22:17:27.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kst8PAvjnErnWGNYi+wS+6pdtKs=', 'Content-Length': '0'} 2023-09-29T22:17:27.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036d8eae75ae918f9-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.305 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.305 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.306 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.306 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.306 DEBUG:boto:Data: 2023-09-29T22:17:27.306 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.306 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.306 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.306 DEBUG:boto:Params: {} 2023-09-29T22:17:27.306 DEBUG:boto:Token: None 2023-09-29T22:17:27.306 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kst8PAvjnErnWGNYi+wS+6pdtKs= 2023-09-29T22:17:27.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kst8PAvjnErnWGNYi+wS+6pdtKs=', 'Content-Length': '0'} 2023-09-29T22:17:27.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9e8a188a7ea466d-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:27.309 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.309 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.309 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.309 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.309 DEBUG:boto:Method: GET 2023-09-29T22:17:27.309 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.309 DEBUG:boto:Data: 2023-09-29T22:17:27.309 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.310 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.310 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.310 DEBUG:boto:Params: {} 2023-09-29T22:17:27.310 DEBUG:boto:Token: None 2023-09-29T22:17:27.310 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.310 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e383befd2ed467a3-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.354 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.355 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:27.355 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.355 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.355 DEBUG:boto:Method: GET 2023-09-29T22:17:27.355 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.355 DEBUG:boto:Data: 2023-09-29T22:17:27.355 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.355 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.355 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.356 DEBUG:boto:Params: {} 2023-09-29T22:17:27.356 DEBUG:boto:Token: None 2023-09-29T22:17:27.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:27.356 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE= 2023-09-29T22:17:27.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UditZkvH3wScH7iu3fHsdLgcspE=', 'Content-Length': '0'} 2023-09-29T22:17:27.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000939f59aab6f4cf65-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.358 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.358 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.358 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.359 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.359 DEBUG:boto:Method: GET 2023-09-29T22:17:27.359 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.359 DEBUG:boto:Data: 2023-09-29T22:17:27.359 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.359 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.359 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.359 DEBUG:boto:Params: {} 2023-09-29T22:17:27.359 DEBUG:boto:Token: None 2023-09-29T22:17:27.359 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.360 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc8cf990a491470-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.362 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.362 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.363 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.363 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.363 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.363 DEBUG:boto:Method: GET 2023-09-29T22:17:27.363 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:27.363 DEBUG:boto:Data: 2023-09-29T22:17:27.363 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.363 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.363 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.363 DEBUG:boto:Params: {} 2023-09-29T22:17:27.364 DEBUG:boto:Token: None 2023-09-29T22:17:27.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.364 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cef3082a2f822bb-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.366 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:27.367 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.367 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.367 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.367 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.367 DEBUG:boto:Method: GET 2023-09-29T22:17:27.368 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.368 DEBUG:boto:Data: 2023-09-29T22:17:27.368 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.368 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.368 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.368 DEBUG:boto:Params: {} 2023-09-29T22:17:27.368 DEBUG:boto:Token: None 2023-09-29T22:17:27.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057ec478152e550b7-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.370 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.370 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.370 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.371 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.371 DEBUG:boto:Method: GET 2023-09-29T22:17:27.371 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.371 DEBUG:boto:Data: 2023-09-29T22:17:27.371 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.371 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.371 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.371 DEBUG:boto:Params: {} 2023-09-29T22:17:27.371 DEBUG:boto:Token: None 2023-09-29T22:17:27.371 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070084f8d231de0ea-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.418 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.419 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.419 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.419 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.419 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.419 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.419 DEBUG:boto:Data: 2023-09-29T22:17:27.419 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.419 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.419 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.419 DEBUG:boto:Params: {} 2023-09-29T22:17:27.420 DEBUG:boto:Token: None 2023-09-29T22:17:27.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F5AiPqvFIZxczw19c936a+DHQLk= 2023-09-29T22:17:27.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F5AiPqvFIZxczw19c936a+DHQLk=', 'Content-Length': '0'} 2023-09-29T22:17:27.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008663b8d3e06a805-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.422 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.422 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.422 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.422 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.422 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.422 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.423 DEBUG:boto:Data: 2023-09-29T22:17:27.423 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.423 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.423 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.423 DEBUG:boto:Params: {} 2023-09-29T22:17:27.423 DEBUG:boto:Token: None 2023-09-29T22:17:27.423 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.423 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F5AiPqvFIZxczw19c936a+DHQLk= 2023-09-29T22:17:27.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F5AiPqvFIZxczw19c936a+DHQLk=', 'Content-Length': '0'} 2023-09-29T22:17:27.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af7fffc03ae1eeb4-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:27.425 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.425 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.425 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.426 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.426 DEBUG:boto:Method: GET 2023-09-29T22:17:27.426 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.426 DEBUG:boto:Data: 2023-09-29T22:17:27.426 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.426 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.426 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.426 DEBUG:boto:Params: {} 2023-09-29T22:17:27.426 DEBUG:boto:Token: None 2023-09-29T22:17:27.427 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.427 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000719299a2d9008edb-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.470 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.471 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.471 DEBUG:boto:Method: GET 2023-09-29T22:17:27.471 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.471 DEBUG:boto:Data: 2023-09-29T22:17:27.471 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.472 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.472 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.472 DEBUG:boto:Params: {} 2023-09-29T22:17:27.472 DEBUG:boto:Token: None 2023-09-29T22:17:27.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058eab6ce4f65e4fe-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.522 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.523 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.523 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.523 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.523 DEBUG:boto:Method: GET 2023-09-29T22:17:27.523 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.523 DEBUG:boto:Data: 2023-09-29T22:17:27.523 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.524 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.524 DEBUG:boto:Params: {} 2023-09-29T22:17:27.524 DEBUG:boto:Token: None 2023-09-29T22:17:27.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000101a34ed10a8e5ac-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.570 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.571 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.571 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.571 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.571 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.571 DEBUG:boto:Method: GET 2023-09-29T22:17:27.571 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.571 DEBUG:boto:Data: 2023-09-29T22:17:27.572 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.572 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.572 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.572 DEBUG:boto:Params: {} 2023-09-29T22:17:27.572 DEBUG:boto:Token: None 2023-09-29T22:17:27.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.572 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f02c98e64198ab-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.575 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.575 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.575 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.576 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.576 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.576 DEBUG:boto:Data: 2023-09-29T22:17:27.576 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.576 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.576 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.576 DEBUG:boto:Params: {} 2023-09-29T22:17:27.576 DEBUG:boto:Token: None 2023-09-29T22:17:27.576 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.576 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I72ZPBdIiI4c07fmGRc5yFks7fc= 2023-09-29T22:17:27.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I72ZPBdIiI4c07fmGRc5yFks7fc=', 'Content-Length': '0'} 2023-09-29T22:17:27.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e07e3d6f7df1477d-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.579 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.579 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.579 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.579 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.579 DEBUG:boto:Data: 2023-09-29T22:17:27.579 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.579 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.579 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.579 DEBUG:boto:Params: {} 2023-09-29T22:17:27.579 DEBUG:boto:Token: None 2023-09-29T22:17:27.580 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.580 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:I72ZPBdIiI4c07fmGRc5yFks7fc= 2023-09-29T22:17:27.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:I72ZPBdIiI4c07fmGRc5yFks7fc=', 'Content-Length': '0'} 2023-09-29T22:17:27.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000371a4d53632cd074-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:27.582 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.582 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.582 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.583 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.583 DEBUG:boto:Method: GET 2023-09-29T22:17:27.583 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.583 DEBUG:boto:Data: 2023-09-29T22:17:27.583 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.583 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.583 DEBUG:boto:Params: {} 2023-09-29T22:17:27.583 DEBUG:boto:Token: None 2023-09-29T22:17:27.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e20b408e994d4eb0-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.626 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.626 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:27.626 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.627 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.627 DEBUG:boto:Method: GET 2023-09-29T22:17:27.627 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.627 DEBUG:boto:Data: 2023-09-29T22:17:27.627 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.627 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.627 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.627 DEBUG:boto:Params: {} 2023-09-29T22:17:27.627 DEBUG:boto:Token: None 2023-09-29T22:17:27.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:27.628 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c= 2023-09-29T22:17:27.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FP0ZBPsf3IAX9bYb4WfSPwn033c=', 'Content-Length': '0'} 2023-09-29T22:17:27.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b65b240159cf64f-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.670 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.670 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.671 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.671 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.671 DEBUG:boto:Method: GET 2023-09-29T22:17:27.671 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.671 DEBUG:boto:Data: 2023-09-29T22:17:27.671 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.671 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.671 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.671 DEBUG:boto:Params: {} 2023-09-29T22:17:27.672 DEBUG:boto:Token: None 2023-09-29T22:17:27.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd6a0562a180254-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.722 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.723 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.723 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.723 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.723 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.723 DEBUG:boto:Method: GET 2023-09-29T22:17:27.723 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:27.724 DEBUG:boto:Data: 2023-09-29T22:17:27.724 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.724 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.724 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.724 DEBUG:boto:Params: {} 2023-09-29T22:17:27.724 DEBUG:boto:Token: None 2023-09-29T22:17:27.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b9bd547f93145d8-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.727 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:27.728 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.728 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.728 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.728 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.728 DEBUG:boto:Method: GET 2023-09-29T22:17:27.728 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.728 DEBUG:boto:Data: 2023-09-29T22:17:27.728 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.728 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.729 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.729 DEBUG:boto:Params: {} 2023-09-29T22:17:27.729 DEBUG:boto:Token: None 2023-09-29T22:17:27.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.729 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaf086b97e7a9aa6-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.731 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.731 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.731 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.731 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.732 DEBUG:boto:Method: GET 2023-09-29T22:17:27.732 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.732 DEBUG:boto:Data: 2023-09-29T22:17:27.732 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.732 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.732 DEBUG:boto:Params: {} 2023-09-29T22:17:27.732 DEBUG:boto:Token: None 2023-09-29T22:17:27.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.733 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008685db1461cee54e-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.734 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.735 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.735 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.735 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.735 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.735 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.735 DEBUG:boto:Data: 2023-09-29T22:17:27.735 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.735 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.735 DEBUG:boto:Params: {} 2023-09-29T22:17:27.736 DEBUG:boto:Token: None 2023-09-29T22:17:27.736 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PGyuY4+jKiA6PLBr/zNrADSOhDw= 2023-09-29T22:17:27.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PGyuY4+jKiA6PLBr/zNrADSOhDw=', 'Content-Length': '0'} 2023-09-29T22:17:27.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa06a0762a0ad85a-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.738 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.738 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.738 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.738 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.738 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.738 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.738 DEBUG:boto:Data: 2023-09-29T22:17:27.738 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.739 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.739 DEBUG:boto:Params: {} 2023-09-29T22:17:27.739 DEBUG:boto:Token: None 2023-09-29T22:17:27.739 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.739 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PGyuY4+jKiA6PLBr/zNrADSOhDw= 2023-09-29T22:17:27.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PGyuY4+jKiA6PLBr/zNrADSOhDw=', 'Content-Length': '0'} 2023-09-29T22:17:27.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f64cadea6f53080-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:27.741 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.741 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.741 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.741 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.741 DEBUG:boto:Method: GET 2023-09-29T22:17:27.741 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.742 DEBUG:boto:Data: 2023-09-29T22:17:27.742 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.742 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.742 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.742 DEBUG:boto:Params: {} 2023-09-29T22:17:27.742 DEBUG:boto:Token: None 2023-09-29T22:17:27.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063139ba12c500d68-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.744 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.744 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.745 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.745 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.745 DEBUG:boto:Method: GET 2023-09-29T22:17:27.745 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.745 DEBUG:boto:Data: 2023-09-29T22:17:27.745 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.745 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.745 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.745 DEBUG:boto:Params: {} 2023-09-29T22:17:27.745 DEBUG:boto:Token: None 2023-09-29T22:17:27.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.746 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038b18df1bef69b67-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.790 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.791 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.791 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.791 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.791 DEBUG:boto:Method: GET 2023-09-29T22:17:27.791 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.791 DEBUG:boto:Data: 2023-09-29T22:17:27.791 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.791 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.791 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.792 DEBUG:boto:Params: {} 2023-09-29T22:17:27.792 DEBUG:boto:Token: None 2023-09-29T22:17:27.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194eeb5e3341c06d-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.795 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.795 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.795 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.795 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.795 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.796 DEBUG:boto:Method: GET 2023-09-29T22:17:27.796 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.796 DEBUG:boto:Data: 2023-09-29T22:17:27.796 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.796 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.796 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.796 DEBUG:boto:Params: {} 2023-09-29T22:17:27.796 DEBUG:boto:Token: None 2023-09-29T22:17:27.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.797 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f20f320b3f042440-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.799 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.799 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.800 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.800 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.800 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.800 DEBUG:boto:Data: 2023-09-29T22:17:27.800 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.800 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.800 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.800 DEBUG:boto:Params: {} 2023-09-29T22:17:27.801 DEBUG:boto:Token: None 2023-09-29T22:17:27.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.801 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ= 2023-09-29T22:17:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4a4cd90acd7b776-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.803 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.803 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.803 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.804 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.804 DEBUG:boto:Data: 2023-09-29T22:17:27.804 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.804 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.804 DEBUG:boto:Params: {} 2023-09-29T22:17:27.804 DEBUG:boto:Token: None 2023-09-29T22:17:27.804 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ= 2023-09-29T22:17:27.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5z54xIorHN5BCCm8fbVOko4WWQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba078ecf2446427d-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:27.807 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.807 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.807 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.807 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.807 DEBUG:boto:Method: GET 2023-09-29T22:17:27.807 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.807 DEBUG:boto:Data: 2023-09-29T22:17:27.808 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.808 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.808 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.808 DEBUG:boto:Params: {} 2023-09-29T22:17:27.808 DEBUG:boto:Token: None 2023-09-29T22:17:27.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.808 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d834598f48f9c03d-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.854 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.855 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:27.855 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.855 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.855 DEBUG:boto:Method: GET 2023-09-29T22:17:27.855 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.855 DEBUG:boto:Data: 2023-09-29T22:17:27.855 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.855 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.856 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.856 DEBUG:boto:Params: {} 2023-09-29T22:17:27.856 DEBUG:boto:Token: None 2023-09-29T22:17:27.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:27.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0= 2023-09-29T22:17:27.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+OV01pWr+RkoUrtWa+pDgFl1j0=', 'Content-Length': '0'} 2023-09-29T22:17:27.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d73fa22a48640d10-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.898 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.899 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.899 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.899 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.899 DEBUG:boto:Method: GET 2023-09-29T22:17:27.899 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.899 DEBUG:boto:Data: 2023-09-29T22:17:27.899 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.899 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.900 DEBUG:boto:Params: {} 2023-09-29T22:17:27.900 DEBUG:boto:Token: None 2023-09-29T22:17:27.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000722aa2b9d199f51d-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.902 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.903 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.903 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.903 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.903 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.903 DEBUG:boto:Method: GET 2023-09-29T22:17:27.903 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:27.904 DEBUG:boto:Data: 2023-09-29T22:17:27.904 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.904 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:27.904 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.904 DEBUG:boto:Params: {} 2023-09-29T22:17:27.904 DEBUG:boto:Token: None 2023-09-29T22:17:27.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY= 2023-09-29T22:17:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoSjAI/7SsfOOp89AeSsSOn+/jY=', 'Content-Length': '0'} 2023-09-29T22:17:27.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e478235b84e5ada-0065174cf7-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.954 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.955 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone3, test-zone1} 2023-09-29T22:17:27.955 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone3, test-zone2} 2023-09-29T22:17:27.955 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.955 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.955 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.956 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:27.956 DEBUG:boto:Data: 2023-09-29T22:17:27.956 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.956 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.956 DEBUG:boto:Params: {} 2023-09-29T22:17:27.956 DEBUG:boto:Token: None 2023-09-29T22:17:27.956 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.957 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY= 2023-09-29T22:17:27.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY=', 'Content-Length': '0'} 2023-09-29T22:17:27.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086402dfd26181761-0065174cf7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.959 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.959 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.959 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.959 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:27.959 DEBUG:boto:Data: 2023-09-29T22:17:27.959 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.960 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.960 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.960 DEBUG:boto:Params: {} 2023-09-29T22:17:27.960 DEBUG:boto:Token: None 2023-09-29T22:17:27.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/ 2023-09-29T22:17:27.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY= 2023-09-29T22:17:27.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YHvd1HpfTpCoAriNkU1tHgfoTTY=', 'Content-Length': '0'} 2023-09-29T22:17:27.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4a0b52d732d6e1e-0065174cf7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.963 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:27.963 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.963 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.963 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.963 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.963 DEBUG:boto:Method: GET 2023-09-29T22:17:27.963 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.963 DEBUG:boto:Data: 2023-09-29T22:17:27.964 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.964 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.964 DEBUG:boto:Params: {} 2023-09-29T22:17:27.964 DEBUG:boto:Token: None 2023-09-29T22:17:27.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03abe552063d93c-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.967 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:27.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:27.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:27.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:27.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:27.968 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.968 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.968 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.968 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.968 DEBUG:boto:Method: GET 2023-09-29T22:17:27.968 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.968 DEBUG:boto:Data: 2023-09-29T22:17:27.969 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.969 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.969 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.969 DEBUG:boto:Params: {} 2023-09-29T22:17:27.969 DEBUG:boto:Token: None 2023-09-29T22:17:27.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909f0722aea1175b-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.972 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.973 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:27.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:27.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:27.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:27.974 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.974 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.975 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.976 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.976 DEBUG:boto:Method: GET 2023-09-29T22:17:27.976 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.976 DEBUG:boto:Data: 2023-09-29T22:17:27.977 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.977 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.977 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.977 DEBUG:boto:Params: {} 2023-09-29T22:17:27.977 DEBUG:boto:Token: None 2023-09-29T22:17:27.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4abd68b88781cec-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.980 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.980 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:27.981 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:27.981 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.981 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.981 DEBUG:boto:Method: GET 2023-09-29T22:17:27.981 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:17:27.981 DEBUG:boto:Data: 2023-09-29T22:17:27.981 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.981 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.981 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.981 DEBUG:boto:Params: {} 2023-09-29T22:17:27.982 DEBUG:boto:Token: None 2023-09-29T22:17:27.982 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/?versions 2023-09-29T22:17:27.982 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w= 2023-09-29T22:17:27.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rNJNp1RiuPtl0nQsXVgK+UbD7/w=', 'Content-Length': '0'} 2023-09-29T22:17:27.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0980602d7090eca-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.984 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:27.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:27.985 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.985 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.985 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.985 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.986 DEBUG:boto:Method: GET 2023-09-29T22:17:27.986 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.986 DEBUG:boto:Data: 2023-09-29T22:17:27.986 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.986 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.986 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.986 DEBUG:boto:Params: {} 2023-09-29T22:17:27.986 DEBUG:boto:Token: None 2023-09-29T22:17:27.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f4ed4dd43b731e1-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.989 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.989 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.989 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.989 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.989 DEBUG:boto:Method: GET 2023-09-29T22:17:27.989 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.990 DEBUG:boto:Data: 2023-09-29T22:17:27.990 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.990 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.990 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.990 DEBUG:boto:Params: {} 2023-09-29T22:17:27.990 DEBUG:boto:Token: None 2023-09-29T22:17:27.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.990 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ= 2023-09-29T22:17:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3iabengTsLaP14kDWzbCNsqjcgQ=', 'Content-Length': '0'} 2023-09-29T22:17:27.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052a689fb9cc0f092-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.993 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.993 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.993 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.993 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.993 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.993 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.993 DEBUG:boto:Data: 2023-09-29T22:17:27.993 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.993 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:27.993 DEBUG:boto:Port: 8001 2023-09-29T22:17:27.994 DEBUG:boto:Params: {} 2023-09-29T22:17:27.994 DEBUG:boto:Token: None 2023-09-29T22:17:27.994 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.994 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw= 2023-09-29T22:17:27.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw=', 'Content-Length': '0'} 2023-09-29T22:17:27.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035877d7ece62fdea-0065174cf7-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.996 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:27.996 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:27.996 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.996 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.996 DEBUG:boto:Method: HEAD 2023-09-29T22:17:27.997 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.997 DEBUG:boto:Data: 2023-09-29T22:17:27.997 DEBUG:boto:Headers: {} 2023-09-29T22:17:27.997 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:27.997 DEBUG:boto:Port: 8000 2023-09-29T22:17:27.997 DEBUG:boto:Params: {} 2023-09-29T22:17:27.997 DEBUG:boto:Token: None 2023-09-29T22:17:27.997 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:27 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:27.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw= 2023-09-29T22:17:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iyQf5xqJiiE00Bf9dow/aHfbYXw=', 'Content-Length': '0'} 2023-09-29T22:17:27.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006388106a65bcaaf9-0065174cf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:27.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:27.999 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.000 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.000 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.000 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.000 DEBUG:boto:Method: GET 2023-09-29T22:17:28.000 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.000 DEBUG:boto:Data: 2023-09-29T22:17:28.000 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.000 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.000 DEBUG:boto:Params: {} 2023-09-29T22:17:28.001 DEBUG:boto:Token: None 2023-09-29T22:17:28.001 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.001 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE= 2023-09-29T22:17:28.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE=', 'Content-Length': '0'} 2023-09-29T22:17:28.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093e75859a317b99f-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.003 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.003 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.003 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.003 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.003 DEBUG:boto:Method: GET 2023-09-29T22:17:28.003 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.003 DEBUG:boto:Data: 2023-09-29T22:17:28.003 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.003 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.003 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.004 DEBUG:boto:Params: {} 2023-09-29T22:17:28.004 DEBUG:boto:Token: None 2023-09-29T22:17:28.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE= 2023-09-29T22:17:28.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE=', 'Content-Length': '0'} 2023-09-29T22:17:28.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af5e28655fe65ee9-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.006 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.006 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.007 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.007 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.007 DEBUG:boto:Method: GET 2023-09-29T22:17:28.007 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.007 DEBUG:boto:Data: 2023-09-29T22:17:28.007 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.007 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.007 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.007 DEBUG:boto:Params: {} 2023-09-29T22:17:28.007 DEBUG:boto:Token: None 2023-09-29T22:17:28.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.008 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f6c957805e0f4c0-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.054 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.055 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.055 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.055 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.055 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.055 DEBUG:boto:Method: GET 2023-09-29T22:17:28.055 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.055 DEBUG:boto:Data: 2023-09-29T22:17:28.055 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.056 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.056 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.056 DEBUG:boto:Params: {} 2023-09-29T22:17:28.056 DEBUG:boto:Token: None 2023-09-29T22:17:28.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6f512684b459490-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.058 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.059 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.059 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.059 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.059 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:28.059 DEBUG:boto:Data: 2023-09-29T22:17:28.059 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.060 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.060 DEBUG:boto:Params: {} 2023-09-29T22:17:28.060 DEBUG:boto:Token: None 2023-09-29T22:17:28.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy 2023-09-29T22:17:28.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n4MT6HPbkoTwKG6lrPIHBVe14gk= 2023-09-29T22:17:28.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n4MT6HPbkoTwKG6lrPIHBVe14gk=', 'Content-Length': '0'} 2023-09-29T22:17:28.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dc31d6f5277e717-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.062 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.063 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.063 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.063 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:17:28.063 DEBUG:boto:Data: 2023-09-29T22:17:28.063 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.063 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.063 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.063 DEBUG:boto:Params: {} 2023-09-29T22:17:28.063 DEBUG:boto:Token: None 2023-09-29T22:17:28.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy 2023-09-29T22:17:28.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n4MT6HPbkoTwKG6lrPIHBVe14gk= 2023-09-29T22:17:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n4MT6HPbkoTwKG6lrPIHBVe14gk=', 'Content-Length': '0'} 2023-09-29T22:17:28.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017e83f694195ab2b-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:28.066 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.066 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.067 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.067 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.067 DEBUG:boto:Method: GET 2023-09-29T22:17:28.067 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.067 DEBUG:boto:Data: 2023-09-29T22:17:28.067 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.067 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.067 DEBUG:boto:Params: {} 2023-09-29T22:17:28.067 DEBUG:boto:Token: None 2023-09-29T22:17:28.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE= 2023-09-29T22:17:28.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE=', 'Content-Length': '0'} 2023-09-29T22:17:28.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d1a4d4a5f27d689-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.069 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:17:28.069 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:17:28.070 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.070 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.070 DEBUG:boto:Method: GET 2023-09-29T22:17:28.070 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.070 DEBUG:boto:Data: 2023-09-29T22:17:28.070 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.070 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.070 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.070 DEBUG:boto:Params: {} 2023-09-29T22:17:28.071 DEBUG:boto:Token: None 2023-09-29T22:17:28.071 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:17:28.071 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE= 2023-09-29T22:17:28.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a5h/FRwrg4ENrAsjyGNLkSVk6RE=', 'Content-Length': '0'} 2023-09-29T22:17:28.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bcf2f8f9f3a6b544-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.073 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.073 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.073 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.073 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.074 DEBUG:boto:Method: GET 2023-09-29T22:17:28.074 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.074 DEBUG:boto:Data: 2023-09-29T22:17:28.074 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.074 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.074 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.074 DEBUG:boto:Params: {} 2023-09-29T22:17:28.074 DEBUG:boto:Token: None 2023-09-29T22:17:28.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b04b53e081b4e26a-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.077 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.077 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.077 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.078 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.078 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.078 DEBUG:boto:Method: GET 2023-09-29T22:17:28.078 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:17:28.078 DEBUG:boto:Data: 2023-09-29T22:17:28.078 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.078 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.078 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.078 DEBUG:boto:Params: {} 2023-09-29T22:17:28.079 DEBUG:boto:Token: None 2023-09-29T22:17:28.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.079 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea11e26fd00e44f5-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.081 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:28.082 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.082 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.082 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.082 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.082 DEBUG:boto:Method: GET 2023-09-29T22:17:28.083 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.083 DEBUG:boto:Data: 2023-09-29T22:17:28.083 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.083 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.083 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.083 DEBUG:boto:Params: {} 2023-09-29T22:17:28.083 DEBUG:boto:Token: None 2023-09-29T22:17:28.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000187ad4f9ee7dd906-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.086 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.086 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.086 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.086 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.086 DEBUG:boto:Method: GET 2023-09-29T22:17:28.086 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.086 DEBUG:boto:Data: 2023-09-29T22:17:28.087 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.087 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.087 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.087 DEBUG:boto:Params: {} 2023-09-29T22:17:28.087 DEBUG:boto:Token: None 2023-09-29T22:17:28.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013b63e58f495e698-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.090 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.090 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.090 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.090 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.090 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.090 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.090 DEBUG:boto:Data: 2023-09-29T22:17:28.090 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.090 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.091 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.091 DEBUG:boto:Params: {} 2023-09-29T22:17:28.091 DEBUG:boto:Token: None 2023-09-29T22:17:28.091 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.091 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZozGb30BxSQ0Vo3VEJDd6FIkJ5s= 2023-09-29T22:17:28.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZozGb30BxSQ0Vo3VEJDd6FIkJ5s=', 'Content-Length': '0'} 2023-09-29T22:17:28.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058a0e241724a4cf0-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.093 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.093 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.094 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.094 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.094 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.094 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.094 DEBUG:boto:Data: 2023-09-29T22:17:28.094 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.094 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.094 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.094 DEBUG:boto:Params: {} 2023-09-29T22:17:28.095 DEBUG:boto:Token: None 2023-09-29T22:17:28.095 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.095 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZozGb30BxSQ0Vo3VEJDd6FIkJ5s= 2023-09-29T22:17:28.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZozGb30BxSQ0Vo3VEJDd6FIkJ5s=', 'Content-Length': '0'} 2023-09-29T22:17:28.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010633460807ad7da-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:28.097 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.097 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.097 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.097 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.097 DEBUG:boto:Method: GET 2023-09-29T22:17:28.098 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.098 DEBUG:boto:Data: 2023-09-29T22:17:28.098 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.098 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.098 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.098 DEBUG:boto:Params: {} 2023-09-29T22:17:28.098 DEBUG:boto:Token: None 2023-09-29T22:17:28.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.098 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b52d1186ef01ed0d-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.142 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.142 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.143 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.143 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.143 DEBUG:boto:Method: GET 2023-09-29T22:17:28.143 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.143 DEBUG:boto:Data: 2023-09-29T22:17:28.143 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.143 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.143 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.143 DEBUG:boto:Params: {} 2023-09-29T22:17:28.143 DEBUG:boto:Token: None 2023-09-29T22:17:28.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0ec85058c72f658-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.187 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.187 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.187 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.187 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.187 DEBUG:boto:Method: GET 2023-09-29T22:17:28.187 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.187 DEBUG:boto:Data: 2023-09-29T22:17:28.187 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.187 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.187 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.188 DEBUG:boto:Params: {} 2023-09-29T22:17:28.188 DEBUG:boto:Token: None 2023-09-29T22:17:28.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.188 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4649d06a1727d9a-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.190 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.191 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.191 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.191 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.191 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.191 DEBUG:boto:Method: GET 2023-09-29T22:17:28.191 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.192 DEBUG:boto:Data: 2023-09-29T22:17:28.192 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.192 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.192 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.192 DEBUG:boto:Params: {} 2023-09-29T22:17:28.192 DEBUG:boto:Token: None 2023-09-29T22:17:28.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.192 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3989bf681eefe29-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.195 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.195 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.195 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.195 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.195 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.196 DEBUG:boto:Data: 2023-09-29T22:17:28.196 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.196 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.196 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.196 DEBUG:boto:Params: {} 2023-09-29T22:17:28.196 DEBUG:boto:Token: None 2023-09-29T22:17:28.196 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.196 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:31rpU4zRL2Y3DLD5quI1WzSF7p4= 2023-09-29T22:17:28.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:31rpU4zRL2Y3DLD5quI1WzSF7p4=', 'Content-Length': '0'} 2023-09-29T22:17:28.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095f989b60b9c55fc-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.198 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.199 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.199 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.199 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.199 DEBUG:boto:Data: 2023-09-29T22:17:28.199 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.199 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.199 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.199 DEBUG:boto:Params: {} 2023-09-29T22:17:28.199 DEBUG:boto:Token: None 2023-09-29T22:17:28.200 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:31rpU4zRL2Y3DLD5quI1WzSF7p4= 2023-09-29T22:17:28.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:31rpU4zRL2Y3DLD5quI1WzSF7p4=', 'Content-Length': '0'} 2023-09-29T22:17:28.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfcb9f93a387aeb4-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:28.202 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.203 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.203 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.203 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.203 DEBUG:boto:Method: GET 2023-09-29T22:17:28.203 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.203 DEBUG:boto:Data: 2023-09-29T22:17:28.203 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.203 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.203 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.203 DEBUG:boto:Params: {} 2023-09-29T22:17:28.204 DEBUG:boto:Token: None 2023-09-29T22:17:28.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069c4fd05e06cea78-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.246 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.247 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:28.247 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.247 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.247 DEBUG:boto:Method: GET 2023-09-29T22:17:28.247 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.247 DEBUG:boto:Data: 2023-09-29T22:17:28.247 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.247 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.247 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.247 DEBUG:boto:Params: {} 2023-09-29T22:17:28.248 DEBUG:boto:Token: None 2023-09-29T22:17:28.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:17:28.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E= 2023-09-29T22:17:28.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9dvj0p3X1Y6RDfEMf033WTPH73E=', 'Content-Length': '0'} 2023-09-29T22:17:28.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e440b0d89535a458-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.290 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.291 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.291 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.291 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.291 DEBUG:boto:Method: GET 2023-09-29T22:17:28.291 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.291 DEBUG:boto:Data: 2023-09-29T22:17:28.291 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.291 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.291 DEBUG:boto:Params: {} 2023-09-29T22:17:28.292 DEBUG:boto:Token: None 2023-09-29T22:17:28.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5eceb1fb2d0d395-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.294 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.295 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.295 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.295 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.295 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.295 DEBUG:boto:Method: GET 2023-09-29T22:17:28.295 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:17:28.296 DEBUG:boto:Data: 2023-09-29T22:17:28.296 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.296 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.296 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.296 DEBUG:boto:Params: {} 2023-09-29T22:17:28.296 DEBUG:boto:Token: None 2023-09-29T22:17:28.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000efc3df3e89e158d-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.299 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:28.299 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.299 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.299 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.300 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.300 DEBUG:boto:Method: GET 2023-09-29T22:17:28.300 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.300 DEBUG:boto:Data: 2023-09-29T22:17:28.300 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.300 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.300 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.300 DEBUG:boto:Params: {} 2023-09-29T22:17:28.300 DEBUG:boto:Token: None 2023-09-29T22:17:28.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.301 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a0a2ef3bcfad814-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.302 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.303 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.303 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.303 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.303 DEBUG:boto:Method: GET 2023-09-29T22:17:28.303 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.303 DEBUG:boto:Data: 2023-09-29T22:17:28.303 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.303 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.303 DEBUG:boto:Params: {} 2023-09-29T22:17:28.304 DEBUG:boto:Token: None 2023-09-29T22:17:28.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9cdaff1f6807dba-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.306 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.306 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.306 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.306 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.306 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.306 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.306 DEBUG:boto:Data: 2023-09-29T22:17:28.307 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.307 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.307 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.307 DEBUG:boto:Params: {} 2023-09-29T22:17:28.307 DEBUG:boto:Token: None 2023-09-29T22:17:28.307 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC6oVjPoqkAK1gjD9nagyGXUe/Y= 2023-09-29T22:17:28.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC6oVjPoqkAK1gjD9nagyGXUe/Y=', 'Content-Length': '0'} 2023-09-29T22:17:28.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2d0c174c35a338d-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.309 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.309 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.310 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.310 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.310 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.310 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.310 DEBUG:boto:Data: 2023-09-29T22:17:28.310 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.310 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.310 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.310 DEBUG:boto:Params: {} 2023-09-29T22:17:28.310 DEBUG:boto:Token: None 2023-09-29T22:17:28.311 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.311 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qC6oVjPoqkAK1gjD9nagyGXUe/Y= 2023-09-29T22:17:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qC6oVjPoqkAK1gjD9nagyGXUe/Y=', 'Content-Length': '0'} 2023-09-29T22:17:28.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c40b3043e4f1d796-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:28.313 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.313 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.313 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.313 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.313 DEBUG:boto:Method: GET 2023-09-29T22:17:28.313 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.313 DEBUG:boto:Data: 2023-09-29T22:17:28.313 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.313 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.314 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.314 DEBUG:boto:Params: {} 2023-09-29T22:17:28.314 DEBUG:boto:Token: None 2023-09-29T22:17:28.314 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.314 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c76ce43ede49529c-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.358 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.359 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.359 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.359 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.359 DEBUG:boto:Method: GET 2023-09-29T22:17:28.359 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.359 DEBUG:boto:Data: 2023-09-29T22:17:28.359 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.359 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.359 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.360 DEBUG:boto:Params: {} 2023-09-29T22:17:28.360 DEBUG:boto:Token: None 2023-09-29T22:17:28.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.360 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039e1220f6f8285a5-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.363 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.363 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.363 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.363 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.363 DEBUG:boto:Method: GET 2023-09-29T22:17:28.363 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.363 DEBUG:boto:Data: 2023-09-29T22:17:28.363 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.363 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.364 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.364 DEBUG:boto:Params: {} 2023-09-29T22:17:28.364 DEBUG:boto:Token: None 2023-09-29T22:17:28.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.364 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f948c28e4aa435d-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.410 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.411 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.411 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.411 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.411 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.411 DEBUG:boto:Method: GET 2023-09-29T22:17:28.412 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.412 DEBUG:boto:Data: 2023-09-29T22:17:28.412 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.412 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.412 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.412 DEBUG:boto:Params: {} 2023-09-29T22:17:28.412 DEBUG:boto:Token: None 2023-09-29T22:17:28.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935b10c1b1e83f1f-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.415 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.415 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.416 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.416 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.416 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.416 DEBUG:boto:Data: 2023-09-29T22:17:28.416 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.416 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.416 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.416 DEBUG:boto:Params: {} 2023-09-29T22:17:28.416 DEBUG:boto:Token: None 2023-09-29T22:17:28.417 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:41cBLwWFwQ19Bib0t0F8/Mi44vc= 2023-09-29T22:17:28.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:41cBLwWFwQ19Bib0t0F8/Mi44vc=', 'Content-Length': '0'} 2023-09-29T22:17:28.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de91dc3d79e09710-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.419 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.419 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.420 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.420 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.420 DEBUG:boto:Data: 2023-09-29T22:17:28.420 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.420 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.420 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.420 DEBUG:boto:Params: {} 2023-09-29T22:17:28.420 DEBUG:boto:Token: None 2023-09-29T22:17:28.421 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:41cBLwWFwQ19Bib0t0F8/Mi44vc= 2023-09-29T22:17:28.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:41cBLwWFwQ19Bib0t0F8/Mi44vc=', 'Content-Length': '0'} 2023-09-29T22:17:28.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbbd319b1516bdce-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:28.423 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.424 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.424 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.424 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.424 DEBUG:boto:Method: GET 2023-09-29T22:17:28.424 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.424 DEBUG:boto:Data: 2023-09-29T22:17:28.424 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.424 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.424 DEBUG:boto:Params: {} 2023-09-29T22:17:28.425 DEBUG:boto:Token: None 2023-09-29T22:17:28.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052dbbf2a66595d5f-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.470 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:28.471 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.471 DEBUG:boto:Method: GET 2023-09-29T22:17:28.471 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.472 DEBUG:boto:Data: 2023-09-29T22:17:28.472 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.472 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.472 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.472 DEBUG:boto:Params: {} 2023-09-29T22:17:28.472 DEBUG:boto:Token: None 2023-09-29T22:17:28.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:17:28.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw= 2023-09-29T22:17:28.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z5Ye/KvUsyeYVd3fkgKSDadeHxw=', 'Content-Length': '0'} 2023-09-29T22:17:28.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003197311b33d5b8f7-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.475 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.475 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.476 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.476 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.476 DEBUG:boto:Method: GET 2023-09-29T22:17:28.476 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.476 DEBUG:boto:Data: 2023-09-29T22:17:28.476 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.477 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.477 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.477 DEBUG:boto:Params: {} 2023-09-29T22:17:28.477 DEBUG:boto:Token: None 2023-09-29T22:17:28.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.477 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aab0715fa12146d-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.522 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.523 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.523 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.523 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.523 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.523 DEBUG:boto:Method: GET 2023-09-29T22:17:28.523 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:17:28.524 DEBUG:boto:Data: 2023-09-29T22:17:28.524 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.524 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.524 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.524 DEBUG:boto:Params: {} 2023-09-29T22:17:28.524 DEBUG:boto:Token: None 2023-09-29T22:17:28.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f314c753e62a94c-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.527 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:28.527 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.528 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.528 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.528 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.528 DEBUG:boto:Method: GET 2023-09-29T22:17:28.528 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.528 DEBUG:boto:Data: 2023-09-29T22:17:28.528 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.528 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.528 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.528 DEBUG:boto:Params: {} 2023-09-29T22:17:28.528 DEBUG:boto:Token: None 2023-09-29T22:17:28.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.529 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453bb78e123206a8-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.533 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.533 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.534 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.534 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.534 DEBUG:boto:Method: GET 2023-09-29T22:17:28.534 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.534 DEBUG:boto:Data: 2023-09-29T22:17:28.534 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.534 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.534 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.534 DEBUG:boto:Params: {} 2023-09-29T22:17:28.534 DEBUG:boto:Token: None 2023-09-29T22:17:28.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.535 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e83fdcb2d068e375-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.537 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.537 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.537 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.537 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.537 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.538 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.538 DEBUG:boto:Data: 2023-09-29T22:17:28.538 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.538 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.538 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.538 DEBUG:boto:Params: {} 2023-09-29T22:17:28.538 DEBUG:boto:Token: None 2023-09-29T22:17:28.538 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.538 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nhHkeAGUAyWSPGyz4gmUnN77nU8= 2023-09-29T22:17:28.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nhHkeAGUAyWSPGyz4gmUnN77nU8=', 'Content-Length': '0'} 2023-09-29T22:17:28.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000440e447bf92c8c3f-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.540 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.540 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.541 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.541 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.541 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.541 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.541 DEBUG:boto:Data: 2023-09-29T22:17:28.541 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.541 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.541 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.541 DEBUG:boto:Params: {} 2023-09-29T22:17:28.541 DEBUG:boto:Token: None 2023-09-29T22:17:28.542 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.542 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nhHkeAGUAyWSPGyz4gmUnN77nU8= 2023-09-29T22:17:28.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nhHkeAGUAyWSPGyz4gmUnN77nU8=', 'Content-Length': '0'} 2023-09-29T22:17:28.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad143d31e32d052d-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:28.544 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.544 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.544 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.544 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.544 DEBUG:boto:Method: GET 2023-09-29T22:17:28.544 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.544 DEBUG:boto:Data: 2023-09-29T22:17:28.544 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.545 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.545 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.545 DEBUG:boto:Params: {} 2023-09-29T22:17:28.545 DEBUG:boto:Token: None 2023-09-29T22:17:28.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.545 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8cfb79d37047d61-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.547 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.547 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.547 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.548 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.548 DEBUG:boto:Method: GET 2023-09-29T22:17:28.548 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.548 DEBUG:boto:Data: 2023-09-29T22:17:28.548 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.548 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.548 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.548 DEBUG:boto:Params: {} 2023-09-29T22:17:28.548 DEBUG:boto:Token: None 2023-09-29T22:17:28.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca95c85934ec6ca-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.594 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.595 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.595 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.595 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.595 DEBUG:boto:Method: GET 2023-09-29T22:17:28.595 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.595 DEBUG:boto:Data: 2023-09-29T22:17:28.595 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.595 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.595 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.596 DEBUG:boto:Params: {} 2023-09-29T22:17:28.596 DEBUG:boto:Token: None 2023-09-29T22:17:28.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039eb93d54abfbaec-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.599 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.599 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.599 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.600 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.600 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.600 DEBUG:boto:Method: GET 2023-09-29T22:17:28.600 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.600 DEBUG:boto:Data: 2023-09-29T22:17:28.600 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.600 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.600 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.600 DEBUG:boto:Params: {} 2023-09-29T22:17:28.600 DEBUG:boto:Token: None 2023-09-29T22:17:28.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.601 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009131c3576f5a9e44-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.604 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.604 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.604 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.604 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.604 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.604 DEBUG:boto:Data: 2023-09-29T22:17:28.604 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.605 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.605 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.605 DEBUG:boto:Params: {} 2023-09-29T22:17:28.605 DEBUG:boto:Token: None 2023-09-29T22:17:28.605 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:50XzK4qNRk3p+/x+r1wBX3EjXy8= 2023-09-29T22:17:28.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:50XzK4qNRk3p+/x+r1wBX3EjXy8=', 'Content-Length': '0'} 2023-09-29T22:17:28.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000883fa155f1f5fc28-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.608 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.608 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.608 DEBUG:boto:Method: HEAD 2023-09-29T22:17:28.608 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.608 DEBUG:boto:Data: 2023-09-29T22:17:28.608 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.608 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.608 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.609 DEBUG:boto:Params: {} 2023-09-29T22:17:28.609 DEBUG:boto:Token: None 2023-09-29T22:17:28.609 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.609 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:50XzK4qNRk3p+/x+r1wBX3EjXy8= 2023-09-29T22:17:28.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:50XzK4qNRk3p+/x+r1wBX3EjXy8=', 'Content-Length': '0'} 2023-09-29T22:17:28.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053c44119a40c3a63-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:28.612 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.612 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.612 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.612 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.612 DEBUG:boto:Method: GET 2023-09-29T22:17:28.612 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.612 DEBUG:boto:Data: 2023-09-29T22:17:28.612 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.612 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.612 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.613 DEBUG:boto:Params: {} 2023-09-29T22:17:28.613 DEBUG:boto:Token: None 2023-09-29T22:17:28.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c3c93d6e92a823b-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.658 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.659 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:28.659 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.659 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.659 DEBUG:boto:Method: GET 2023-09-29T22:17:28.659 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.659 DEBUG:boto:Data: 2023-09-29T22:17:28.659 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.659 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.659 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.660 DEBUG:boto:Params: {} 2023-09-29T22:17:28.660 DEBUG:boto:Token: None 2023-09-29T22:17:28.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:17:28.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q= 2023-09-29T22:17:28.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uvDgzrI9yFzfo14uzglXz64xT1Q=', 'Content-Length': '0'} 2023-09-29T22:17:28.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfe306a9e40c6147-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.702 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.703 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.703 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.703 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.703 DEBUG:boto:Method: GET 2023-09-29T22:17:28.703 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.703 DEBUG:boto:Data: 2023-09-29T22:17:28.703 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:28.703 DEBUG:boto:Port: 8001 2023-09-29T22:17:28.704 DEBUG:boto:Params: {} 2023-09-29T22:17:28.704 DEBUG:boto:Token: None 2023-09-29T22:17:28.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f8fbb7ecca412d-0065174cf8-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.707 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.707 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:28.707 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:28.707 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.707 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.707 DEBUG:boto:Method: GET 2023-09-29T22:17:28.707 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:17:28.708 DEBUG:boto:Data: 2023-09-29T22:17:28.708 DEBUG:boto:Headers: {} 2023-09-29T22:17:28.708 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:28.708 DEBUG:boto:Port: 8000 2023-09-29T22:17:28.708 DEBUG:boto:Params: {} 2023-09-29T22:17:28.708 DEBUG:boto:Token: None 2023-09-29T22:17:28.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:28 GMT /dmuhsu-17/ 2023-09-29T22:17:28.708 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA= 2023-09-29T22:17:28.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqv5ddP4Hf55nndjWY6Zw0BwlA=', 'Content-Length': '0'} 2023-09-29T22:17:28.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba1e6ea2df6c3037-0065174cf8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:28.711 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:28.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone3, test-zone2} 2023-09-29T22:17:28.711 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-18 2023-09-29T22:17:28.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.712 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:28.771 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:28.771 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:28.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:28.809+0000 7f1da4529880 1 bucket sync caught up with source: 2023-09-29T22:17:28.810 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.42.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:17:28.810 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.42.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:17:28.810 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:28.809+0000 7f1da4529880 0 bucket checkpoint complete 2023-09-29T22:17:28.824 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-18 2023-09-29T22:17:28.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.825 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:28.884 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:28.884 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:28.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:28.921+0000 7fc1feb1f880 1 bucket sync caught up with source: 2023-09-29T22:17:28.921 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.42.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:17:28.921 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.42.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:17:28.922 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:28.921+0000 7fc1feb1f880 0 bucket checkpoint complete 2023-09-29T22:17:28.935 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-18 2023-09-29T22:17:28.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:28.936 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:28.989 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:28.990 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:29.022 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:29.022+0000 7f26b340f880 1 bucket sync caught up with source: 2023-09-29T22:17:29.023 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000010.24.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:17:29.023 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000010.24.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:17:29.023 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:29.022+0000 7f26b340f880 0 bucket checkpoint complete 2023-09-29T22:17:29.034 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-18 2023-09-29T22:17:29.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:29.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:29.035 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:29.087 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:29.087 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:29.120 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:29.121+0000 7fe49b00a880 1 bucket sync caught up with source: 2023-09-29T22:17:29.121 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000010.24.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:17:29.121 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000010.24.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:17:29.121 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:29.121+0000 7fe49b00a880 0 bucket checkpoint complete 2023-09-29T22:17:29.133 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-18 2023-09-29T22:17:29.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:29.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:29.134 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:29.184 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:29.185 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:29.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:29.217+0000 7fa1d3fdc880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:29.217 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000014.57.10, , , , , ] 2023-09-29T22:17:29.217 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:34.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:34.218+0000 7fa1d3fdc880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:34.218 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000014.57.10, , , , , ] 2023-09-29T22:17:34.219 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:39.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:39.220+0000 7fa1d3fdc880 1 bucket sync caught up with source: 2023-09-29T22:17:39.220 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:39.220 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:39.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:39.220+0000 7fa1d3fdc880 0 bucket checkpoint complete 2023-09-29T22:17:39.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-18 2023-09-29T22:17:39.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:39.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:39.236 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:39.295 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:39.296 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:39.335 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:39.336+0000 7fb7d9f24880 1 bucket sync caught up with source: 2023-09-29T22:17:39.336 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:39.336 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000016.61.10, , , , , ] 2023-09-29T22:17:39.336 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:39.336+0000 7fb7d9f24880 0 bucket checkpoint complete 2023-09-29T22:17:39.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone2, test-zone1} 2023-09-29T22:17:39.349 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.349 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.349 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.349 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:39.349 DEBUG:boto:Data: 2023-09-29T22:17:39.349 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.349 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.349 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.349 DEBUG:boto:Params: {} 2023-09-29T22:17:39.350 DEBUG:boto:Token: None 2023-09-29T22:17:39.350 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wHTLnleSmGMjyC5vC8dGpaBNqAg= 2023-09-29T22:17:39.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wHTLnleSmGMjyC5vC8dGpaBNqAg=', 'Content-Length': '0'} 2023-09-29T22:17:39.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089a7c28c230e3935-0065174d03-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.353 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.353 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.353 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.353 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:39.353 DEBUG:boto:Data: 2023-09-29T22:17:39.353 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.353 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.353 DEBUG:boto:Params: {} 2023-09-29T22:17:39.354 DEBUG:boto:Token: None 2023-09-29T22:17:39.354 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.354 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wHTLnleSmGMjyC5vC8dGpaBNqAg= 2023-09-29T22:17:39.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wHTLnleSmGMjyC5vC8dGpaBNqAg=', 'Content-Length': '0'} 2023-09-29T22:17:39.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c8c762f4115ee2b-0065174d03-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:39.357 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.357 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.357 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.357 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.357 DEBUG:boto:Method: GET 2023-09-29T22:17:39.357 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.357 DEBUG:boto:Data: 2023-09-29T22:17:39.357 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.357 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.358 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.358 DEBUG:boto:Params: {} 2023-09-29T22:17:39.358 DEBUG:boto:Token: None 2023-09-29T22:17:39.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/?versions 2023-09-29T22:17:39.358 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g= 2023-09-29T22:17:39.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g=', 'Content-Length': '0'} 2023-09-29T22:17:39.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc92e926efc9cd03-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.360 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.361 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:39.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:39.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:39.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:39.362 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:39.362 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.362 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.362 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.362 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.362 DEBUG:boto:Method: GET 2023-09-29T22:17:39.362 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.362 DEBUG:boto:Data: 2023-09-29T22:17:39.362 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.363 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.363 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.363 DEBUG:boto:Params: {} 2023-09-29T22:17:39.363 DEBUG:boto:Token: None 2023-09-29T22:17:39.363 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/?versions 2023-09-29T22:17:39.363 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g= 2023-09-29T22:17:39.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g=', 'Content-Length': '0'} 2023-09-29T22:17:39.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b755ce71e78c63d-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.366 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:39.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:39.367 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.367 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.367 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.367 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.367 DEBUG:boto:Method: GET 2023-09-29T22:17:39.367 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.367 DEBUG:boto:Data: 2023-09-29T22:17:39.367 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.367 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.368 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.368 DEBUG:boto:Params: {} 2023-09-29T22:17:39.368 DEBUG:boto:Token: None 2023-09-29T22:17:39.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/?versions 2023-09-29T22:17:39.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g= 2023-09-29T22:17:39.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g=', 'Content-Length': '0'} 2023-09-29T22:17:39.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a34fad12fab182f-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.370 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.371 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.371 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.371 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.371 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.371 DEBUG:boto:Method: GET 2023-09-29T22:17:39.371 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:39.371 DEBUG:boto:Data: 2023-09-29T22:17:39.371 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.372 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.372 DEBUG:boto:Params: {} 2023-09-29T22:17:39.372 DEBUG:boto:Token: None 2023-09-29T22:17:39.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/?versions 2023-09-29T22:17:39.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g= 2023-09-29T22:17:39.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nbRnK+VbExrvEwVnYRmaQELVZ1g=', 'Content-Length': '0'} 2023-09-29T22:17:39.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039aa2edac12c5ac0-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.375 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:39.375 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.376 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.376 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.376 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.376 DEBUG:boto:Method: GET 2023-09-29T22:17:39.376 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.376 DEBUG:boto:Data: 2023-09-29T22:17:39.376 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.376 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.376 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.376 DEBUG:boto:Params: {} 2023-09-29T22:17:39.377 DEBUG:boto:Token: None 2023-09-29T22:17:39.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.377 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1c575ee2e704874-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.379 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.379 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.379 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.379 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.379 DEBUG:boto:Method: GET 2023-09-29T22:17:39.379 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.380 DEBUG:boto:Data: 2023-09-29T22:17:39.380 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.380 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.380 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.380 DEBUG:boto:Params: {} 2023-09-29T22:17:39.380 DEBUG:boto:Token: None 2023-09-29T22:17:39.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c47cf74ebed42aa-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.382 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.383 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.383 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.383 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.383 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.383 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.383 DEBUG:boto:Data: 2023-09-29T22:17:39.383 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.383 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.383 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.383 DEBUG:boto:Params: {} 2023-09-29T22:17:39.384 DEBUG:boto:Token: None 2023-09-29T22:17:39.384 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.384 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GX4zSdsAAqVcoe1trvPsWjRO9qU= 2023-09-29T22:17:39.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GX4zSdsAAqVcoe1trvPsWjRO9qU=', 'Content-Length': '0'} 2023-09-29T22:17:39.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de64e50aea624ac9-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.386 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.386 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.386 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.386 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.386 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.387 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.387 DEBUG:boto:Data: 2023-09-29T22:17:39.387 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.387 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.387 DEBUG:boto:Params: {} 2023-09-29T22:17:39.387 DEBUG:boto:Token: None 2023-09-29T22:17:39.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.387 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GX4zSdsAAqVcoe1trvPsWjRO9qU= 2023-09-29T22:17:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GX4zSdsAAqVcoe1trvPsWjRO9qU=', 'Content-Length': '0'} 2023-09-29T22:17:39.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d84ee5e8e6781eb7-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:39.389 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.390 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.390 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.390 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.390 DEBUG:boto:Method: GET 2023-09-29T22:17:39.390 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.390 DEBUG:boto:Data: 2023-09-29T22:17:39.390 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.390 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.390 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.390 DEBUG:boto:Params: {} 2023-09-29T22:17:39.391 DEBUG:boto:Token: None 2023-09-29T22:17:39.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.391 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb81c2cd3a34977d-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.393 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.393 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.393 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.393 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.393 DEBUG:boto:Method: GET 2023-09-29T22:17:39.393 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.393 DEBUG:boto:Data: 2023-09-29T22:17:39.394 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.394 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.394 DEBUG:boto:Params: {} 2023-09-29T22:17:39.394 DEBUG:boto:Token: None 2023-09-29T22:17:39.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.394 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073e59b3e7ed19ab0-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.396 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.396 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.397 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.397 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.397 DEBUG:boto:Method: GET 2023-09-29T22:17:39.397 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.397 DEBUG:boto:Data: 2023-09-29T22:17:39.397 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.397 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.397 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.397 DEBUG:boto:Params: {} 2023-09-29T22:17:39.397 DEBUG:boto:Token: None 2023-09-29T22:17:39.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.398 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb52d070e69dd72e-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.400 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.401 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.401 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.401 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.401 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.401 DEBUG:boto:Method: GET 2023-09-29T22:17:39.401 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.401 DEBUG:boto:Data: 2023-09-29T22:17:39.402 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.402 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.402 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.402 DEBUG:boto:Params: {} 2023-09-29T22:17:39.402 DEBUG:boto:Token: None 2023-09-29T22:17:39.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da6e1597f8e2767a-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.407 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.408 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.408 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.408 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.408 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:39.408 DEBUG:boto:Data: 2023-09-29T22:17:39.408 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.409 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.409 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.409 DEBUG:boto:Params: {} 2023-09-29T22:17:39.409 DEBUG:boto:Token: None 2023-09-29T22:17:39.409 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy 2023-09-29T22:17:39.409 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7UwSrNwBCmN3RWQo9mEGtJvwoNs= 2023-09-29T22:17:39.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7UwSrNwBCmN3RWQo9mEGtJvwoNs=', 'Content-Length': '0'} 2023-09-29T22:17:39.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a15f4d49ea1ba675-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.411 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.411 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.412 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.412 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:39.412 DEBUG:boto:Data: 2023-09-29T22:17:39.412 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.412 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.412 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.412 DEBUG:boto:Params: {} 2023-09-29T22:17:39.412 DEBUG:boto:Token: None 2023-09-29T22:17:39.412 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy 2023-09-29T22:17:39.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7UwSrNwBCmN3RWQo9mEGtJvwoNs= 2023-09-29T22:17:39.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7UwSrNwBCmN3RWQo9mEGtJvwoNs=', 'Content-Length': '0'} 2023-09-29T22:17:39.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc0e6afedceecc82-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:39.415 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.415 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.415 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.415 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.415 DEBUG:boto:Method: GET 2023-09-29T22:17:39.416 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.416 DEBUG:boto:Data: 2023-09-29T22:17:39.416 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.416 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.416 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.416 DEBUG:boto:Params: {} 2023-09-29T22:17:39.416 DEBUG:boto:Token: None 2023-09-29T22:17:39.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003759f570889a4017-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.419 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:39.419 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:39.419 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.419 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.419 DEBUG:boto:Method: GET 2023-09-29T22:17:39.419 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.419 DEBUG:boto:Data: 2023-09-29T22:17:39.419 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.419 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.419 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.420 DEBUG:boto:Params: {} 2023-09-29T22:17:39.420 DEBUG:boto:Token: None 2023-09-29T22:17:39.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:39.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM= 2023-09-29T22:17:39.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tNRQs46LEOS8HTU2GM5mHHTWaM=', 'Content-Length': '0'} 2023-09-29T22:17:39.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015896d900ccab35d-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.422 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.422 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.422 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.422 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.422 DEBUG:boto:Method: GET 2023-09-29T22:17:39.423 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.423 DEBUG:boto:Data: 2023-09-29T22:17:39.423 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.423 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.423 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.423 DEBUG:boto:Params: {} 2023-09-29T22:17:39.423 DEBUG:boto:Token: None 2023-09-29T22:17:39.423 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.423 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ead09cbaa0a0326-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.470 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.471 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.471 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.471 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.472 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.472 DEBUG:boto:Method: GET 2023-09-29T22:17:39.472 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:39.472 DEBUG:boto:Data: 2023-09-29T22:17:39.472 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.472 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.472 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.473 DEBUG:boto:Params: {} 2023-09-29T22:17:39.473 DEBUG:boto:Token: None 2023-09-29T22:17:39.473 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6615cac6ad04c3-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.476 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:39.477 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.477 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.477 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.477 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.477 DEBUG:boto:Method: GET 2023-09-29T22:17:39.477 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.477 DEBUG:boto:Data: 2023-09-29T22:17:39.478 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.478 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.478 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.478 DEBUG:boto:Params: {} 2023-09-29T22:17:39.478 DEBUG:boto:Token: None 2023-09-29T22:17:39.478 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.478 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001abc0baa7453fbdb-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.522 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.523 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.523 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.523 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.523 DEBUG:boto:Method: GET 2023-09-29T22:17:39.523 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.523 DEBUG:boto:Data: 2023-09-29T22:17:39.523 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.524 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.524 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.524 DEBUG:boto:Params: {} 2023-09-29T22:17:39.524 DEBUG:boto:Token: None 2023-09-29T22:17:39.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000761abb0ad1727391-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.527 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.527 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.527 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.527 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.527 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.527 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.528 DEBUG:boto:Data: 2023-09-29T22:17:39.528 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.528 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.528 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.528 DEBUG:boto:Params: {} 2023-09-29T22:17:39.528 DEBUG:boto:Token: None 2023-09-29T22:17:39.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.528 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NBNHBWKY9rI3wNF3n+eSafu5ww= 2023-09-29T22:17:39.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NBNHBWKY9rI3wNF3n+eSafu5ww=', 'Content-Length': '0'} 2023-09-29T22:17:39.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042233137205ca955-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.531 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.531 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.531 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.531 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.531 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.531 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.531 DEBUG:boto:Data: 2023-09-29T22:17:39.531 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.532 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.532 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.532 DEBUG:boto:Params: {} 2023-09-29T22:17:39.532 DEBUG:boto:Token: None 2023-09-29T22:17:39.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NBNHBWKY9rI3wNF3n+eSafu5ww= 2023-09-29T22:17:39.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NBNHBWKY9rI3wNF3n+eSafu5ww=', 'Content-Length': '0'} 2023-09-29T22:17:39.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be1dfda6cd7a9779-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:39.535 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.535 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.535 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.535 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.535 DEBUG:boto:Method: GET 2023-09-29T22:17:39.535 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.535 DEBUG:boto:Data: 2023-09-29T22:17:39.536 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.536 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.536 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.536 DEBUG:boto:Params: {} 2023-09-29T22:17:39.536 DEBUG:boto:Token: None 2023-09-29T22:17:39.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.537 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1e3e7266b2bf63a-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.582 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.583 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.583 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.583 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.583 DEBUG:boto:Method: GET 2023-09-29T22:17:39.583 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.583 DEBUG:boto:Data: 2023-09-29T22:17:39.583 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.584 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.584 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.584 DEBUG:boto:Params: {} 2023-09-29T22:17:39.584 DEBUG:boto:Token: None 2023-09-29T22:17:39.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004009ea7915dafa42-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.587 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.587 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.587 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.587 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.587 DEBUG:boto:Method: GET 2023-09-29T22:17:39.588 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.588 DEBUG:boto:Data: 2023-09-29T22:17:39.588 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.588 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.588 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.588 DEBUG:boto:Params: {} 2023-09-29T22:17:39.588 DEBUG:boto:Token: None 2023-09-29T22:17:39.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.589 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef48be7d64afa1f-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.591 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.591 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.592 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.592 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.592 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.592 DEBUG:boto:Method: GET 2023-09-29T22:17:39.592 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.592 DEBUG:boto:Data: 2023-09-29T22:17:39.592 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.592 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.593 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.593 DEBUG:boto:Params: {} 2023-09-29T22:17:39.593 DEBUG:boto:Token: None 2023-09-29T22:17:39.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.593 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5cc718f8b4ab19-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.642 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.643 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.643 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.643 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.643 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.643 DEBUG:boto:Data: 2023-09-29T22:17:39.643 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.644 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.644 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.644 DEBUG:boto:Params: {} 2023-09-29T22:17:39.644 DEBUG:boto:Token: None 2023-09-29T22:17:39.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sB+TVax7tkPAGuE4N45vxs+H4T8= 2023-09-29T22:17:39.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sB+TVax7tkPAGuE4N45vxs+H4T8=', 'Content-Length': '0'} 2023-09-29T22:17:39.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f17a72cff62969f4-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.647 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.647 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.648 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.648 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.648 DEBUG:boto:Data: 2023-09-29T22:17:39.648 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.648 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.649 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.649 DEBUG:boto:Params: {} 2023-09-29T22:17:39.649 DEBUG:boto:Token: None 2023-09-29T22:17:39.649 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.649 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sB+TVax7tkPAGuE4N45vxs+H4T8= 2023-09-29T22:17:39.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sB+TVax7tkPAGuE4N45vxs+H4T8=', 'Content-Length': '0'} 2023-09-29T22:17:39.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fb98b573df186fd-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:39.653 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.653 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.653 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.653 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.653 DEBUG:boto:Method: GET 2023-09-29T22:17:39.653 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.654 DEBUG:boto:Data: 2023-09-29T22:17:39.654 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.654 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.654 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.654 DEBUG:boto:Params: {} 2023-09-29T22:17:39.654 DEBUG:boto:Token: None 2023-09-29T22:17:39.654 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a993b8f61c5c143d-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.698 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.698 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:39.699 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.699 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.699 DEBUG:boto:Method: GET 2023-09-29T22:17:39.699 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.699 DEBUG:boto:Data: 2023-09-29T22:17:39.699 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.699 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.700 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.700 DEBUG:boto:Params: {} 2023-09-29T22:17:39.700 DEBUG:boto:Token: None 2023-09-29T22:17:39.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:39.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U= 2023-09-29T22:17:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jnrEDJJBketnS8rZF/aOT1q0a3U=', 'Content-Length': '0'} 2023-09-29T22:17:39.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1a45a42d3b4d1b6-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.703 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.703 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.703 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.703 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.704 DEBUG:boto:Method: GET 2023-09-29T22:17:39.704 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.704 DEBUG:boto:Data: 2023-09-29T22:17:39.704 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.704 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.704 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.704 DEBUG:boto:Params: {} 2023-09-29T22:17:39.705 DEBUG:boto:Token: None 2023-09-29T22:17:39.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.705 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f28a611eebb6bae1-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.750 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.751 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.751 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.751 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.751 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.752 DEBUG:boto:Method: GET 2023-09-29T22:17:39.752 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:39.752 DEBUG:boto:Data: 2023-09-29T22:17:39.752 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.752 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.752 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.752 DEBUG:boto:Params: {} 2023-09-29T22:17:39.752 DEBUG:boto:Token: None 2023-09-29T22:17:39.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.753 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d0f52f03de177aa-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.756 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:39.756 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.756 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.757 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.757 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.757 DEBUG:boto:Method: GET 2023-09-29T22:17:39.757 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.757 DEBUG:boto:Data: 2023-09-29T22:17:39.757 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.757 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.757 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.758 DEBUG:boto:Params: {} 2023-09-29T22:17:39.758 DEBUG:boto:Token: None 2023-09-29T22:17:39.758 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.758 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008b9309f4969b58e-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.761 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.761 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.761 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.761 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.761 DEBUG:boto:Method: GET 2023-09-29T22:17:39.761 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.761 DEBUG:boto:Data: 2023-09-29T22:17:39.761 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.762 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.762 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.762 DEBUG:boto:Params: {} 2023-09-29T22:17:39.762 DEBUG:boto:Token: None 2023-09-29T22:17:39.762 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.762 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033fe18020dbcaf34-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.810 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.811 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.811 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.811 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.811 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.811 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.811 DEBUG:boto:Data: 2023-09-29T22:17:39.811 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.811 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.811 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.811 DEBUG:boto:Params: {} 2023-09-29T22:17:39.813 DEBUG:boto:Token: None 2023-09-29T22:17:39.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.813 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VD+SIUr10WH5YrD3leqDPS66PRk= 2023-09-29T22:17:39.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VD+SIUr10WH5YrD3leqDPS66PRk=', 'Content-Length': '0'} 2023-09-29T22:17:39.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9ae57fa636e7027-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.815 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.815 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.815 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.815 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.815 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.816 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.816 DEBUG:boto:Data: 2023-09-29T22:17:39.816 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.816 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.816 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.816 DEBUG:boto:Params: {} 2023-09-29T22:17:39.816 DEBUG:boto:Token: None 2023-09-29T22:17:39.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VD+SIUr10WH5YrD3leqDPS66PRk= 2023-09-29T22:17:39.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VD+SIUr10WH5YrD3leqDPS66PRk=', 'Content-Length': '0'} 2023-09-29T22:17:39.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b2ffb1fc98ca18-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:39.819 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.819 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.819 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.819 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.819 DEBUG:boto:Method: GET 2023-09-29T22:17:39.819 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.819 DEBUG:boto:Data: 2023-09-29T22:17:39.819 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.820 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.820 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.820 DEBUG:boto:Params: {} 2023-09-29T22:17:39.820 DEBUG:boto:Token: None 2023-09-29T22:17:39.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000663b838281817c54-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.822 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.822 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.822 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.822 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.823 DEBUG:boto:Method: GET 2023-09-29T22:17:39.823 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.823 DEBUG:boto:Data: 2023-09-29T22:17:39.823 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.823 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.823 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.823 DEBUG:boto:Params: {} 2023-09-29T22:17:39.823 DEBUG:boto:Token: None 2023-09-29T22:17:39.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.823 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000369669f0bfb1adf4-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.866 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.866 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.867 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.867 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.867 DEBUG:boto:Method: GET 2023-09-29T22:17:39.867 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.867 DEBUG:boto:Data: 2023-09-29T22:17:39.867 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.867 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.867 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.867 DEBUG:boto:Params: {} 2023-09-29T22:17:39.867 DEBUG:boto:Token: None 2023-09-29T22:17:39.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.868 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa6a3470bba424a-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.870 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.871 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.871 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.871 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.871 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.871 DEBUG:boto:Method: GET 2023-09-29T22:17:39.871 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.871 DEBUG:boto:Data: 2023-09-29T22:17:39.872 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.872 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.872 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.872 DEBUG:boto:Params: {} 2023-09-29T22:17:39.872 DEBUG:boto:Token: None 2023-09-29T22:17:39.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2ba25f0eb5e6c1-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.875 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.875 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.875 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.875 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.875 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.876 DEBUG:boto:Data: 2023-09-29T22:17:39.876 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.876 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.876 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.876 DEBUG:boto:Params: {} 2023-09-29T22:17:39.876 DEBUG:boto:Token: None 2023-09-29T22:17:39.876 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EDN1Ei2t4Kfux/3tZWUJvoC5yXw= 2023-09-29T22:17:39.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EDN1Ei2t4Kfux/3tZWUJvoC5yXw=', 'Content-Length': '0'} 2023-09-29T22:17:39.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000030afb0affa376f3-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.879 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.879 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.879 DEBUG:boto:Method: HEAD 2023-09-29T22:17:39.879 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.879 DEBUG:boto:Data: 2023-09-29T22:17:39.879 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.879 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.879 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.879 DEBUG:boto:Params: {} 2023-09-29T22:17:39.880 DEBUG:boto:Token: None 2023-09-29T22:17:39.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EDN1Ei2t4Kfux/3tZWUJvoC5yXw= 2023-09-29T22:17:39.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EDN1Ei2t4Kfux/3tZWUJvoC5yXw=', 'Content-Length': '0'} 2023-09-29T22:17:39.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064c948c5226d8791-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:39.883 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.883 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.883 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.883 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.883 DEBUG:boto:Method: GET 2023-09-29T22:17:39.883 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.883 DEBUG:boto:Data: 2023-09-29T22:17:39.883 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.883 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.883 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.883 DEBUG:boto:Params: {} 2023-09-29T22:17:39.884 DEBUG:boto:Token: None 2023-09-29T22:17:39.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cd5218845c6a54f-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.926 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.927 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:39.927 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.927 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.927 DEBUG:boto:Method: GET 2023-09-29T22:17:39.927 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.927 DEBUG:boto:Data: 2023-09-29T22:17:39.927 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.927 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.927 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.927 DEBUG:boto:Params: {} 2023-09-29T22:17:39.927 DEBUG:boto:Token: None 2023-09-29T22:17:39.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:39.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw= 2023-09-29T22:17:39.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wrdHEqwoYJgTxt3JQBJA2ojO1Bw=', 'Content-Length': '0'} 2023-09-29T22:17:39.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a0e58dc3f9f9626-0065174d03-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.970 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.970 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.971 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.971 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.971 DEBUG:boto:Method: GET 2023-09-29T22:17:39.971 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.971 DEBUG:boto:Data: 2023-09-29T22:17:39.971 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.971 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:39.971 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.971 DEBUG:boto:Params: {} 2023-09-29T22:17:39.971 DEBUG:boto:Token: None 2023-09-29T22:17:39.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be747c982970de21-0065174d03-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:39.974 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:39.975 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:39.975 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:39.975 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.975 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.975 DEBUG:boto:Method: GET 2023-09-29T22:17:39.975 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:39.975 DEBUG:boto:Data: 2023-09-29T22:17:39.975 DEBUG:boto:Headers: {} 2023-09-29T22:17:39.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:39.975 DEBUG:boto:Port: 8000 2023-09-29T22:17:39.976 DEBUG:boto:Params: {} 2023-09-29T22:17:39.976 DEBUG:boto:Token: None 2023-09-29T22:17:39.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:39 GMT /dmuhsu-18/ 2023-09-29T22:17:39.976 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs= 2023-09-29T22:17:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:39 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VquR7Wlbl0yXRWoqVhiSc+ARYFs=', 'Content-Length': '0'} 2023-09-29T22:17:39.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7b3bdbdf414282-0065174d03-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.026 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:40.027 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.027 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.027 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.027 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.027 DEBUG:boto:Method: GET 2023-09-29T22:17:40.027 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.028 DEBUG:boto:Data: 2023-09-29T22:17:40.028 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.028 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.028 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.028 DEBUG:boto:Params: {} 2023-09-29T22:17:40.028 DEBUG:boto:Token: None 2023-09-29T22:17:40.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069597fd6404b2a29-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.031 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.031 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.031 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.031 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.031 DEBUG:boto:Method: GET 2023-09-29T22:17:40.031 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.031 DEBUG:boto:Data: 2023-09-29T22:17:40.031 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.032 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.032 DEBUG:boto:Params: {} 2023-09-29T22:17:40.032 DEBUG:boto:Token: None 2023-09-29T22:17:40.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.032 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025b31d850de24169-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.034 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.034 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.035 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.035 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.035 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.035 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.035 DEBUG:boto:Data: 2023-09-29T22:17:40.035 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.035 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.035 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.035 DEBUG:boto:Params: {} 2023-09-29T22:17:40.035 DEBUG:boto:Token: None 2023-09-29T22:17:40.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.036 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NeRmYCrsFmTjb9/pK7TixwTopXI= 2023-09-29T22:17:40.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NeRmYCrsFmTjb9/pK7TixwTopXI=', 'Content-Length': '0'} 2023-09-29T22:17:40.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba44922d6ea255e7-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.038 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.038 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.038 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.038 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.038 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.038 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.038 DEBUG:boto:Data: 2023-09-29T22:17:40.038 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.039 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.039 DEBUG:boto:Params: {} 2023-09-29T22:17:40.039 DEBUG:boto:Token: None 2023-09-29T22:17:40.039 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.039 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NeRmYCrsFmTjb9/pK7TixwTopXI= 2023-09-29T22:17:40.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NeRmYCrsFmTjb9/pK7TixwTopXI=', 'Content-Length': '0'} 2023-09-29T22:17:40.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a92f028975e5d2f9-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:40.041 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.041 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.041 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.041 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.042 DEBUG:boto:Method: GET 2023-09-29T22:17:40.042 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.042 DEBUG:boto:Data: 2023-09-29T22:17:40.042 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.042 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.042 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.042 DEBUG:boto:Params: {} 2023-09-29T22:17:40.042 DEBUG:boto:Token: None 2023-09-29T22:17:40.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fcd2c25cee7efe8-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.086 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.086 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.087 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.087 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.087 DEBUG:boto:Method: GET 2023-09-29T22:17:40.087 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.087 DEBUG:boto:Data: 2023-09-29T22:17:40.087 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.087 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.087 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.087 DEBUG:boto:Params: {} 2023-09-29T22:17:40.088 DEBUG:boto:Token: None 2023-09-29T22:17:40.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d0a4e040ae7eec6-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.134 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.135 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.135 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.135 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.135 DEBUG:boto:Method: GET 2023-09-29T22:17:40.135 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.135 DEBUG:boto:Data: 2023-09-29T22:17:40.135 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.135 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.135 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.136 DEBUG:boto:Params: {} 2023-09-29T22:17:40.136 DEBUG:boto:Token: None 2023-09-29T22:17:40.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.136 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000992df1a7c9f33b3a-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.139 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.139 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.139 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.139 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.139 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.140 DEBUG:boto:Method: GET 2023-09-29T22:17:40.140 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.140 DEBUG:boto:Data: 2023-09-29T22:17:40.140 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.140 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.140 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.140 DEBUG:boto:Params: {} 2023-09-29T22:17:40.140 DEBUG:boto:Token: None 2023-09-29T22:17:40.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a7ef7f3c1c41e7-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.190 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.191 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.191 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.191 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.191 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.191 DEBUG:boto:Data: 2023-09-29T22:17:40.191 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.192 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.192 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.192 DEBUG:boto:Params: {} 2023-09-29T22:17:40.192 DEBUG:boto:Token: None 2023-09-29T22:17:40.193 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lWquzKvWoOaOQl3lcuKsqq62oz4= 2023-09-29T22:17:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lWquzKvWoOaOQl3lcuKsqq62oz4=', 'Content-Length': '0'} 2023-09-29T22:17:40.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfe6369c6681447a-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.196 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.196 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.196 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.196 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.197 DEBUG:boto:Data: 2023-09-29T22:17:40.197 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.197 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.197 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.197 DEBUG:boto:Params: {} 2023-09-29T22:17:40.197 DEBUG:boto:Token: None 2023-09-29T22:17:40.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lWquzKvWoOaOQl3lcuKsqq62oz4= 2023-09-29T22:17:40.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lWquzKvWoOaOQl3lcuKsqq62oz4=', 'Content-Length': '0'} 2023-09-29T22:17:40.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2fe7359b2978d48-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:40.200 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.200 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.200 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.200 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.200 DEBUG:boto:Method: GET 2023-09-29T22:17:40.201 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.201 DEBUG:boto:Data: 2023-09-29T22:17:40.201 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.201 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.201 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.201 DEBUG:boto:Params: {} 2023-09-29T22:17:40.201 DEBUG:boto:Token: None 2023-09-29T22:17:40.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.202 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc4d88769b422c55-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.246 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.246 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:40.247 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.247 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.247 DEBUG:boto:Method: GET 2023-09-29T22:17:40.247 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.247 DEBUG:boto:Data: 2023-09-29T22:17:40.247 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.247 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.247 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.247 DEBUG:boto:Params: {} 2023-09-29T22:17:40.248 DEBUG:boto:Token: None 2023-09-29T22:17:40.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:40.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08= 2023-09-29T22:17:40.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SxU7z41MzX6IXpcSRVW0FStmq08=', 'Content-Length': '0'} 2023-09-29T22:17:40.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f8cf9328245e63a-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.294 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.295 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.295 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.295 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.295 DEBUG:boto:Method: GET 2023-09-29T22:17:40.295 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.295 DEBUG:boto:Data: 2023-09-29T22:17:40.295 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.295 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:40.295 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.295 DEBUG:boto:Params: {} 2023-09-29T22:17:40.296 DEBUG:boto:Token: None 2023-09-29T22:17:40.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb2813317b9d63f-0065174d04-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.299 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.299 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.299 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.299 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.299 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.300 DEBUG:boto:Method: GET 2023-09-29T22:17:40.300 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:40.300 DEBUG:boto:Data: 2023-09-29T22:17:40.300 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.300 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.300 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.300 DEBUG:boto:Params: {} 2023-09-29T22:17:40.300 DEBUG:boto:Token: None 2023-09-29T22:17:40.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.301 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d8d71cbcc20a5b5-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.303 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.304 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone2, test-zone1} 2023-09-29T22:17:40.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone3, test-zone1} 2023-09-29T22:17:40.304 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.304 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.304 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.304 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:40.305 DEBUG:boto:Data: 2023-09-29T22:17:40.305 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.305 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.305 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.305 DEBUG:boto:Params: {} 2023-09-29T22:17:40.305 DEBUG:boto:Token: None 2023-09-29T22:17:40.305 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.305 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7E8M/YRIuJ0C7UHEcG46NskBpUM= 2023-09-29T22:17:40.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7E8M/YRIuJ0C7UHEcG46NskBpUM=', 'Content-Length': '0'} 2023-09-29T22:17:40.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6e5ed98070f4553-0065174d04-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.308 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.308 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.308 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.308 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:40.308 DEBUG:boto:Data: 2023-09-29T22:17:40.308 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.309 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.309 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.309 DEBUG:boto:Params: {} 2023-09-29T22:17:40.309 DEBUG:boto:Token: None 2023-09-29T22:17:40.309 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.309 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7E8M/YRIuJ0C7UHEcG46NskBpUM= 2023-09-29T22:17:40.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7E8M/YRIuJ0C7UHEcG46NskBpUM=', 'Content-Length': '0'} 2023-09-29T22:17:40.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7d3b4cd78bb3c3e-0065174d04-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:40.312 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.312 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.312 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.312 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.312 DEBUG:boto:Method: GET 2023-09-29T22:17:40.312 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.312 DEBUG:boto:Data: 2023-09-29T22:17:40.312 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.312 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.313 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.313 DEBUG:boto:Params: {} 2023-09-29T22:17:40.313 DEBUG:boto:Token: None 2023-09-29T22:17:40.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/?versions 2023-09-29T22:17:40.313 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA= 2023-09-29T22:17:40.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA=', 'Content-Length': '0'} 2023-09-29T22:17:40.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f6fc28d46dd074c-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.315 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:40.316 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:40.316 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.317 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.317 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.317 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.317 DEBUG:boto:Method: GET 2023-09-29T22:17:40.317 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.317 DEBUG:boto:Data: 2023-09-29T22:17:40.317 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.317 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.317 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.317 DEBUG:boto:Params: {} 2023-09-29T22:17:40.318 DEBUG:boto:Token: None 2023-09-29T22:17:40.318 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/?versions 2023-09-29T22:17:40.318 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA= 2023-09-29T22:17:40.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA=', 'Content-Length': '0'} 2023-09-29T22:17:40.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143396ac3d0429bc-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.320 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.321 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:40.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:40.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:40.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:40.321 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.321 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.322 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.322 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.322 DEBUG:boto:Method: GET 2023-09-29T22:17:40.322 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.322 DEBUG:boto:Data: 2023-09-29T22:17:40.322 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.322 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.322 DEBUG:boto:Params: {} 2023-09-29T22:17:40.323 DEBUG:boto:Token: None 2023-09-29T22:17:40.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/?versions 2023-09-29T22:17:40.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA= 2023-09-29T22:17:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA=', 'Content-Length': '0'} 2023-09-29T22:17:40.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4d569d1dada87ea-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.325 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.326 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.326 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.326 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.326 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.326 DEBUG:boto:Method: GET 2023-09-29T22:17:40.326 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:40.327 DEBUG:boto:Data: 2023-09-29T22:17:40.327 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.327 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.327 DEBUG:boto:Params: {} 2023-09-29T22:17:40.327 DEBUG:boto:Token: None 2023-09-29T22:17:40.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/?versions 2023-09-29T22:17:40.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA= 2023-09-29T22:17:40.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nxaXZp2A33avvsSUXZ0QG1UkDNA=', 'Content-Length': '0'} 2023-09-29T22:17:40.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008976823660e5dfcc-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.330 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:40.331 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.331 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.331 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.331 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.331 DEBUG:boto:Method: GET 2023-09-29T22:17:40.331 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.331 DEBUG:boto:Data: 2023-09-29T22:17:40.331 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.331 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.332 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.332 DEBUG:boto:Params: {} 2023-09-29T22:17:40.332 DEBUG:boto:Token: None 2023-09-29T22:17:40.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.332 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d2d2d33ed3202ca-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.334 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.334 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.334 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.334 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.335 DEBUG:boto:Method: GET 2023-09-29T22:17:40.335 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.335 DEBUG:boto:Data: 2023-09-29T22:17:40.335 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.335 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.335 DEBUG:boto:Params: {} 2023-09-29T22:17:40.335 DEBUG:boto:Token: None 2023-09-29T22:17:40.335 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000054ebb22d2850690-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.338 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.338 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.338 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.338 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.338 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.338 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.338 DEBUG:boto:Data: 2023-09-29T22:17:40.338 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.338 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.339 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.339 DEBUG:boto:Params: {} 2023-09-29T22:17:40.339 DEBUG:boto:Token: None 2023-09-29T22:17:40.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.339 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lpiFzQz66iUNR0QCbmp61XsS7mQ= 2023-09-29T22:17:40.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lpiFzQz66iUNR0QCbmp61XsS7mQ=', 'Content-Length': '0'} 2023-09-29T22:17:40.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e5e3d2c208cec2b-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.341 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.341 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.342 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.342 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.342 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.342 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.342 DEBUG:boto:Data: 2023-09-29T22:17:40.342 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.342 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.342 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.342 DEBUG:boto:Params: {} 2023-09-29T22:17:40.343 DEBUG:boto:Token: None 2023-09-29T22:17:40.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.343 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lpiFzQz66iUNR0QCbmp61XsS7mQ= 2023-09-29T22:17:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lpiFzQz66iUNR0QCbmp61XsS7mQ=', 'Content-Length': '0'} 2023-09-29T22:17:40.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d5a7a9a88e54263-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:40.345 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.345 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.345 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.345 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.346 DEBUG:boto:Method: GET 2023-09-29T22:17:40.346 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.346 DEBUG:boto:Data: 2023-09-29T22:17:40.346 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.346 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.346 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.346 DEBUG:boto:Params: {} 2023-09-29T22:17:40.346 DEBUG:boto:Token: None 2023-09-29T22:17:40.346 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062a8a6c9521abe8a-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.348 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.349 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.349 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.349 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.349 DEBUG:boto:Method: GET 2023-09-29T22:17:40.349 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.349 DEBUG:boto:Data: 2023-09-29T22:17:40.349 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.349 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.349 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.349 DEBUG:boto:Params: {} 2023-09-29T22:17:40.350 DEBUG:boto:Token: None 2023-09-29T22:17:40.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048d0332a165e996c-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.352 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.352 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.352 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.352 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.352 DEBUG:boto:Method: GET 2023-09-29T22:17:40.352 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.353 DEBUG:boto:Data: 2023-09-29T22:17:40.353 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.353 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.353 DEBUG:boto:Params: {} 2023-09-29T22:17:40.353 DEBUG:boto:Token: None 2023-09-29T22:17:40.353 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.353 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a71125a0113b8c9-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.356 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.356 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.356 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.356 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.356 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.357 DEBUG:boto:Method: GET 2023-09-29T22:17:40.357 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.357 DEBUG:boto:Data: 2023-09-29T22:17:40.357 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.357 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.357 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.357 DEBUG:boto:Params: {} 2023-09-29T22:17:40.357 DEBUG:boto:Token: None 2023-09-29T22:17:40.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa98958d84a67695-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.402 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.403 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.403 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.403 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.403 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:40.403 DEBUG:boto:Data: 2023-09-29T22:17:40.403 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.403 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.403 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.404 DEBUG:boto:Params: {} 2023-09-29T22:17:40.404 DEBUG:boto:Token: None 2023-09-29T22:17:40.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy 2023-09-29T22:17:40.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9vVyyG1c2YldXAdKLhZTDB++BqA= 2023-09-29T22:17:40.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9vVyyG1c2YldXAdKLhZTDB++BqA=', 'Content-Length': '0'} 2023-09-29T22:17:40.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081bb2cb0c7fb8d21-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.406 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.406 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.406 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.407 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:40.407 DEBUG:boto:Data: 2023-09-29T22:17:40.407 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.407 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.407 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.407 DEBUG:boto:Params: {} 2023-09-29T22:17:40.407 DEBUG:boto:Token: None 2023-09-29T22:17:40.407 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy 2023-09-29T22:17:40.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9vVyyG1c2YldXAdKLhZTDB++BqA= 2023-09-29T22:17:40.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9vVyyG1c2YldXAdKLhZTDB++BqA=', 'Content-Length': '0'} 2023-09-29T22:17:40.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008620474ec9d41d4f-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:40.410 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.410 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.410 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.410 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.411 DEBUG:boto:Method: GET 2023-09-29T22:17:40.411 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.411 DEBUG:boto:Data: 2023-09-29T22:17:40.411 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.411 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.411 DEBUG:boto:Params: {} 2023-09-29T22:17:40.411 DEBUG:boto:Token: None 2023-09-29T22:17:40.411 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f806fe1d4615f4a-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.413 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:40.414 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:40.414 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.414 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.414 DEBUG:boto:Method: GET 2023-09-29T22:17:40.414 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.414 DEBUG:boto:Data: 2023-09-29T22:17:40.414 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.414 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.414 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.415 DEBUG:boto:Params: {} 2023-09-29T22:17:40.415 DEBUG:boto:Token: None 2023-09-29T22:17:40.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:40.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw= 2023-09-29T22:17:40.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+YsV0DhvcrWqQPaQfj0qnkOmLTw=', 'Content-Length': '0'} 2023-09-29T22:17:40.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1bbaee58ac3560c-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.417 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.417 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.417 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.417 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.417 DEBUG:boto:Method: GET 2023-09-29T22:17:40.418 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.418 DEBUG:boto:Data: 2023-09-29T22:17:40.418 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.418 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.418 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.418 DEBUG:boto:Params: {} 2023-09-29T22:17:40.418 DEBUG:boto:Token: None 2023-09-29T22:17:40.418 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.418 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a445f4194826a06-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.423 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.423 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.423 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.424 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.424 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.424 DEBUG:boto:Method: GET 2023-09-29T22:17:40.424 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:40.424 DEBUG:boto:Data: 2023-09-29T22:17:40.424 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.424 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.424 DEBUG:boto:Params: {} 2023-09-29T22:17:40.424 DEBUG:boto:Token: None 2023-09-29T22:17:40.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000241984f0fe6868fd-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.427 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:40.428 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.428 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.428 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.428 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.428 DEBUG:boto:Method: GET 2023-09-29T22:17:40.428 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.428 DEBUG:boto:Data: 2023-09-29T22:17:40.428 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.428 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.428 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.428 DEBUG:boto:Params: {} 2023-09-29T22:17:40.429 DEBUG:boto:Token: None 2023-09-29T22:17:40.429 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.429 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004557a137e2d54e43-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.478 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.479 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.479 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.479 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.479 DEBUG:boto:Method: GET 2023-09-29T22:17:40.479 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.479 DEBUG:boto:Data: 2023-09-29T22:17:40.479 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.479 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.479 DEBUG:boto:Params: {} 2023-09-29T22:17:40.480 DEBUG:boto:Token: None 2023-09-29T22:17:40.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e64a89eeb62cdd8-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.482 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.482 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.482 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.483 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.483 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.483 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.483 DEBUG:boto:Data: 2023-09-29T22:17:40.483 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.483 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.483 DEBUG:boto:Params: {} 2023-09-29T22:17:40.483 DEBUG:boto:Token: None 2023-09-29T22:17:40.483 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nrfQU4mmezGaEZM6l9eG5objR2o= 2023-09-29T22:17:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nrfQU4mmezGaEZM6l9eG5objR2o=', 'Content-Length': '0'} 2023-09-29T22:17:40.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000608e95e4dfececd5-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.486 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.486 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.486 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.486 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.486 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.486 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.486 DEBUG:boto:Data: 2023-09-29T22:17:40.486 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.486 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.487 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.487 DEBUG:boto:Params: {} 2023-09-29T22:17:40.487 DEBUG:boto:Token: None 2023-09-29T22:17:40.487 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.487 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nrfQU4mmezGaEZM6l9eG5objR2o= 2023-09-29T22:17:40.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nrfQU4mmezGaEZM6l9eG5objR2o=', 'Content-Length': '0'} 2023-09-29T22:17:40.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3f27f1dfb28d49f-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:40.489 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.489 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.489 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.489 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.489 DEBUG:boto:Method: GET 2023-09-29T22:17:40.490 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.490 DEBUG:boto:Data: 2023-09-29T22:17:40.490 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.490 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.490 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.490 DEBUG:boto:Params: {} 2023-09-29T22:17:40.490 DEBUG:boto:Token: None 2023-09-29T22:17:40.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ce1defbce67c4e5-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.534 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.535 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.535 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.535 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.535 DEBUG:boto:Method: GET 2023-09-29T22:17:40.535 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.535 DEBUG:boto:Data: 2023-09-29T22:17:40.535 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.535 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.536 DEBUG:boto:Params: {} 2023-09-29T22:17:40.536 DEBUG:boto:Token: None 2023-09-29T22:17:40.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df171e665140b273-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.538 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.538 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.539 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.539 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.539 DEBUG:boto:Method: GET 2023-09-29T22:17:40.539 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.539 DEBUG:boto:Data: 2023-09-29T22:17:40.539 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.539 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.539 DEBUG:boto:Params: {} 2023-09-29T22:17:40.540 DEBUG:boto:Token: None 2023-09-29T22:17:40.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003537df3ff9650262-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.542 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.543 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.543 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.543 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.543 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.543 DEBUG:boto:Method: GET 2023-09-29T22:17:40.543 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.543 DEBUG:boto:Data: 2023-09-29T22:17:40.543 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.544 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.544 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.544 DEBUG:boto:Params: {} 2023-09-29T22:17:40.544 DEBUG:boto:Token: None 2023-09-29T22:17:40.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116a02c1480e23f2-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.547 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.548 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.548 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.548 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.548 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.548 DEBUG:boto:Data: 2023-09-29T22:17:40.548 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.548 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.548 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.548 DEBUG:boto:Params: {} 2023-09-29T22:17:40.549 DEBUG:boto:Token: None 2023-09-29T22:17:40.549 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sdLUeNaSTfQMDoJASIiwOAh+eCM= 2023-09-29T22:17:40.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sdLUeNaSTfQMDoJASIiwOAh+eCM=', 'Content-Length': '0'} 2023-09-29T22:17:40.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000511015f0a9e1c2c3-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.551 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.551 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.551 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.551 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.551 DEBUG:boto:Data: 2023-09-29T22:17:40.552 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.552 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.552 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.552 DEBUG:boto:Params: {} 2023-09-29T22:17:40.552 DEBUG:boto:Token: None 2023-09-29T22:17:40.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sdLUeNaSTfQMDoJASIiwOAh+eCM= 2023-09-29T22:17:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sdLUeNaSTfQMDoJASIiwOAh+eCM=', 'Content-Length': '0'} 2023-09-29T22:17:40.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043da2925c01f6188-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:40.556 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.556 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.556 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.556 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.556 DEBUG:boto:Method: GET 2023-09-29T22:17:40.556 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.556 DEBUG:boto:Data: 2023-09-29T22:17:40.556 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.556 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.556 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.557 DEBUG:boto:Params: {} 2023-09-29T22:17:40.557 DEBUG:boto:Token: None 2023-09-29T22:17:40.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.557 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000569d201cf0db7ed2-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.602 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.603 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:40.603 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.603 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.603 DEBUG:boto:Method: GET 2023-09-29T22:17:40.603 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.603 DEBUG:boto:Data: 2023-09-29T22:17:40.603 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.603 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.603 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.604 DEBUG:boto:Params: {} 2023-09-29T22:17:40.604 DEBUG:boto:Token: None 2023-09-29T22:17:40.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:40.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w= 2023-09-29T22:17:40.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xzhkk2L0UZmFYxyWWjuGBL7XY8w=', 'Content-Length': '0'} 2023-09-29T22:17:40.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052c8576472a30c0f-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.606 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.606 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.606 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.606 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.607 DEBUG:boto:Method: GET 2023-09-29T22:17:40.607 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.607 DEBUG:boto:Data: 2023-09-29T22:17:40.607 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.607 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.607 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.607 DEBUG:boto:Params: {} 2023-09-29T22:17:40.607 DEBUG:boto:Token: None 2023-09-29T22:17:40.607 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.607 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000365a83bc87944509-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.610 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.611 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.611 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.611 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.611 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.611 DEBUG:boto:Method: GET 2023-09-29T22:17:40.611 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:40.611 DEBUG:boto:Data: 2023-09-29T22:17:40.611 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.611 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.611 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.612 DEBUG:boto:Params: {} 2023-09-29T22:17:40.612 DEBUG:boto:Token: None 2023-09-29T22:17:40.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.612 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008333911ad02ad347-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.614 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:40.615 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.615 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.615 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.615 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.615 DEBUG:boto:Method: GET 2023-09-29T22:17:40.615 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.616 DEBUG:boto:Data: 2023-09-29T22:17:40.616 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.616 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.616 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.616 DEBUG:boto:Params: {} 2023-09-29T22:17:40.616 DEBUG:boto:Token: None 2023-09-29T22:17:40.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000647ecb57eeb67ec3-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.619 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.619 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.619 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.619 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.619 DEBUG:boto:Method: GET 2023-09-29T22:17:40.619 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.619 DEBUG:boto:Data: 2023-09-29T22:17:40.619 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.620 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.620 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.620 DEBUG:boto:Params: {} 2023-09-29T22:17:40.620 DEBUG:boto:Token: None 2023-09-29T22:17:40.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000daff0b1fd06f924-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.662 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.710 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.711 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.711 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.711 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.711 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.711 DEBUG:boto:Data: 2023-09-29T22:17:40.711 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.711 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.711 DEBUG:boto:Params: {} 2023-09-29T22:17:40.712 DEBUG:boto:Token: None 2023-09-29T22:17:40.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VOPwUJRKtXceSMZUIMpqKsuUW10= 2023-09-29T22:17:40.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VOPwUJRKtXceSMZUIMpqKsuUW10=', 'Content-Length': '0'} 2023-09-29T22:17:40.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eab7006f7e247b8-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.714 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.714 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.714 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.714 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.715 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.715 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.715 DEBUG:boto:Data: 2023-09-29T22:17:40.715 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.715 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.715 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.715 DEBUG:boto:Params: {} 2023-09-29T22:17:40.715 DEBUG:boto:Token: None 2023-09-29T22:17:40.715 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.715 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VOPwUJRKtXceSMZUIMpqKsuUW10= 2023-09-29T22:17:40.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VOPwUJRKtXceSMZUIMpqKsuUW10=', 'Content-Length': '0'} 2023-09-29T22:17:40.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033ba79f96052a6fa-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:40.717 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.718 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.718 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.718 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.718 DEBUG:boto:Method: GET 2023-09-29T22:17:40.718 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.718 DEBUG:boto:Data: 2023-09-29T22:17:40.718 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.718 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.718 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.719 DEBUG:boto:Params: {} 2023-09-29T22:17:40.719 DEBUG:boto:Token: None 2023-09-29T22:17:40.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085d9e93633566072-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.762 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.829 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.829 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.830 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.830 DEBUG:boto:Method: GET 2023-09-29T22:17:40.830 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.830 DEBUG:boto:Data: 2023-09-29T22:17:40.830 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.830 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.830 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.830 DEBUG:boto:Params: {} 2023-09-29T22:17:40.831 DEBUG:boto:Token: None 2023-09-29T22:17:40.831 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000958328f4d5e19ab9-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.878 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.879 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.879 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.879 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.879 DEBUG:boto:Method: GET 2023-09-29T22:17:40.879 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.879 DEBUG:boto:Data: 2023-09-29T22:17:40.879 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.879 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.880 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.880 DEBUG:boto:Params: {} 2023-09-29T22:17:40.880 DEBUG:boto:Token: None 2023-09-29T22:17:40.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b9399310023c04-0065174d04-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.883 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.883 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:40.883 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:40.884 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.884 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.884 DEBUG:boto:Method: GET 2023-09-29T22:17:40.884 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:40.884 DEBUG:boto:Data: 2023-09-29T22:17:40.884 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.884 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.884 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.884 DEBUG:boto:Params: {} 2023-09-29T22:17:40.885 DEBUG:boto:Token: None 2023-09-29T22:17:40.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/ 2023-09-29T22:17:40.885 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78= 2023-09-29T22:17:40.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6GD/k06wEDb0kv89xCIlFkiVC78=', 'Content-Length': '0'} 2023-09-29T22:17:40.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ba902615b4d9bc-0065174d04-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.934 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:40.935 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.935 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.935 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.935 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.935 DEBUG:boto:Data: 2023-09-29T22:17:40.935 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.935 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.936 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.936 DEBUG:boto:Params: {} 2023-09-29T22:17:40.936 DEBUG:boto:Token: None 2023-09-29T22:17:40.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/ZdseavTGNdY4M2i66x5N294HSQ= 2023-09-29T22:17:40.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZdseavTGNdY4M2i66x5N294HSQ=', 'Content-Length': '0'} 2023-09-29T22:17:40.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d52307dc8944736-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.939 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.939 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.939 DEBUG:boto:Method: HEAD 2023-09-29T22:17:40.939 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.939 DEBUG:boto:Data: 2023-09-29T22:17:40.939 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.939 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.939 DEBUG:boto:Params: {} 2023-09-29T22:17:40.940 DEBUG:boto:Token: None 2023-09-29T22:17:40.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/ZdseavTGNdY4M2i66x5N294HSQ= 2023-09-29T22:17:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZdseavTGNdY4M2i66x5N294HSQ=', 'Content-Length': '0'} 2023-09-29T22:17:40.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034042d7bac7c10d9-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:40.943 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.943 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.943 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.943 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.943 DEBUG:boto:Method: GET 2023-09-29T22:17:40.943 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.943 DEBUG:boto:Data: 2023-09-29T22:17:40.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:40.944 DEBUG:boto:Port: 8001 2023-09-29T22:17:40.944 DEBUG:boto:Params: {} 2023-09-29T22:17:40.944 DEBUG:boto:Token: None 2023-09-29T22:17:40.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049c60268a6797df9-0065174d04-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:40.986 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.986 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:40.987 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.987 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.987 DEBUG:boto:Method: GET 2023-09-29T22:17:40.987 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.987 DEBUG:boto:Data: 2023-09-29T22:17:40.987 DEBUG:boto:Headers: {} 2023-09-29T22:17:40.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:40.988 DEBUG:boto:Port: 8000 2023-09-29T22:17:40.988 DEBUG:boto:Params: {} 2023-09-29T22:17:40.988 DEBUG:boto:Token: None 2023-09-29T22:17:40.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:40 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:40.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8= 2023-09-29T22:17:40.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z6ZtCJpbBrtdo5wD1nbUxtG84J8=', 'Content-Length': '0'} 2023-09-29T22:17:40.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dc2a208132f4b43-0065174d04-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.030 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.030 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.030 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.031 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.031 DEBUG:boto:Method: GET 2023-09-29T22:17:41.031 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.031 DEBUG:boto:Data: 2023-09-29T22:17:41.031 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.031 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.031 DEBUG:boto:Params: {} 2023-09-29T22:17:41.032 DEBUG:boto:Token: None 2023-09-29T22:17:41.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.032 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe958a161d21bd1-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.035 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.035 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.035 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.036 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.037 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.037 DEBUG:boto:Method: GET 2023-09-29T22:17:41.037 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.037 DEBUG:boto:Data: 2023-09-29T22:17:41.037 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.038 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.038 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.038 DEBUG:boto:Params: {} 2023-09-29T22:17:41.038 DEBUG:boto:Token: None 2023-09-29T22:17:41.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3d2a4e1656e268-0065174d05-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.041 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:41.042 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.042 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.042 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.042 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.042 DEBUG:boto:Method: GET 2023-09-29T22:17:41.042 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.042 DEBUG:boto:Data: 2023-09-29T22:17:41.043 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.043 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.043 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.043 DEBUG:boto:Params: {} 2023-09-29T22:17:41.043 DEBUG:boto:Token: None 2023-09-29T22:17:41.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002f14541502e8c74-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.046 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.046 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.046 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.046 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.046 DEBUG:boto:Method: GET 2023-09-29T22:17:41.046 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.046 DEBUG:boto:Data: 2023-09-29T22:17:41.047 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.047 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.047 DEBUG:boto:Params: {} 2023-09-29T22:17:41.047 DEBUG:boto:Token: None 2023-09-29T22:17:41.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009d2694f02b6298f-0065174d05-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.050 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.050 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.050 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.050 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.051 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.051 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.051 DEBUG:boto:Data: 2023-09-29T22:17:41.051 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.051 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.051 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.051 DEBUG:boto:Params: {} 2023-09-29T22:17:41.051 DEBUG:boto:Token: None 2023-09-29T22:17:41.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D8jMdBE4Z8Fr9UhifNo1bVGIN/8= 2023-09-29T22:17:41.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D8jMdBE4Z8Fr9UhifNo1bVGIN/8=', 'Content-Length': '0'} 2023-09-29T22:17:41.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001baeb001e490449f-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.054 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.054 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.054 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.054 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.055 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.055 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.055 DEBUG:boto:Data: 2023-09-29T22:17:41.055 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.055 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.055 DEBUG:boto:Params: {} 2023-09-29T22:17:41.055 DEBUG:boto:Token: None 2023-09-29T22:17:41.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D8jMdBE4Z8Fr9UhifNo1bVGIN/8= 2023-09-29T22:17:41.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D8jMdBE4Z8Fr9UhifNo1bVGIN/8=', 'Content-Length': '0'} 2023-09-29T22:17:41.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5dfc6e5357a08a6-0065174d05-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:41.058 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.058 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.058 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.058 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.058 DEBUG:boto:Method: GET 2023-09-29T22:17:41.058 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.059 DEBUG:boto:Data: 2023-09-29T22:17:41.059 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.059 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.059 DEBUG:boto:Params: {} 2023-09-29T22:17:41.059 DEBUG:boto:Token: None 2023-09-29T22:17:41.059 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.059 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b6457b903e5c6b1-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.061 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.062 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.062 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.062 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.062 DEBUG:boto:Method: GET 2023-09-29T22:17:41.062 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.062 DEBUG:boto:Data: 2023-09-29T22:17:41.063 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.063 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.063 DEBUG:boto:Params: {} 2023-09-29T22:17:41.063 DEBUG:boto:Token: None 2023-09-29T22:17:41.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.063 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c181d9c5e1fc5a90-0065174d05-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.106 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.107 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.107 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.107 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.107 DEBUG:boto:Method: GET 2023-09-29T22:17:41.107 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.107 DEBUG:boto:Data: 2023-09-29T22:17:41.107 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.107 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.108 DEBUG:boto:Params: {} 2023-09-29T22:17:41.108 DEBUG:boto:Token: None 2023-09-29T22:17:41.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f9ffd72631a340-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.111 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.111 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.111 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.111 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.111 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.112 DEBUG:boto:Method: GET 2023-09-29T22:17:41.112 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.112 DEBUG:boto:Data: 2023-09-29T22:17:41.112 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.112 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.112 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.112 DEBUG:boto:Params: {} 2023-09-29T22:17:41.112 DEBUG:boto:Token: None 2023-09-29T22:17:41.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007080ec377e266a2f-0065174d05-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.162 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.163 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.163 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.163 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.164 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.164 DEBUG:boto:Data: 2023-09-29T22:17:41.164 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.164 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.164 DEBUG:boto:Params: {} 2023-09-29T22:17:41.164 DEBUG:boto:Token: None 2023-09-29T22:17:41.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sa7YWxjJ0jEb90LWFhyXqd6qN2M= 2023-09-29T22:17:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sa7YWxjJ0jEb90LWFhyXqd6qN2M=', 'Content-Length': '0'} 2023-09-29T22:17:41.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82da01fd624177b-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.168 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.168 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.168 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.168 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.168 DEBUG:boto:Data: 2023-09-29T22:17:41.168 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.168 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.169 DEBUG:boto:Params: {} 2023-09-29T22:17:41.169 DEBUG:boto:Token: None 2023-09-29T22:17:41.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sa7YWxjJ0jEb90LWFhyXqd6qN2M= 2023-09-29T22:17:41.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sa7YWxjJ0jEb90LWFhyXqd6qN2M=', 'Content-Length': '0'} 2023-09-29T22:17:41.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e48d18e6449d596-0065174d05-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:41.172 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.172 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.172 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.172 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.172 DEBUG:boto:Method: GET 2023-09-29T22:17:41.172 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.172 DEBUG:boto:Data: 2023-09-29T22:17:41.172 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.173 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.173 DEBUG:boto:Params: {} 2023-09-29T22:17:41.173 DEBUG:boto:Token: None 2023-09-29T22:17:41.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.173 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c1a487932b0951a-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.222 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.223 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:41.223 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.223 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.223 DEBUG:boto:Method: GET 2023-09-29T22:17:41.223 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.223 DEBUG:boto:Data: 2023-09-29T22:17:41.223 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.223 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.224 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.224 DEBUG:boto:Params: {} 2023-09-29T22:17:41.224 DEBUG:boto:Token: None 2023-09-29T22:17:41.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:41.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4= 2023-09-29T22:17:41.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0g3TPM9IC/kR8MHjZWMMFaOOmc4=', 'Content-Length': '0'} 2023-09-29T22:17:41.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8c2dc3fdf0bd954-0065174d05-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.270 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.271 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.271 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.271 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.271 DEBUG:boto:Method: GET 2023-09-29T22:17:41.271 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.271 DEBUG:boto:Data: 2023-09-29T22:17:41.271 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.271 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.272 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.272 DEBUG:boto:Params: {} 2023-09-29T22:17:41.272 DEBUG:boto:Token: None 2023-09-29T22:17:41.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4b341f736ca315-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.275 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.275 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.275 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.275 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.276 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.276 DEBUG:boto:Method: GET 2023-09-29T22:17:41.276 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:41.276 DEBUG:boto:Data: 2023-09-29T22:17:41.276 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.276 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:41.276 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.276 DEBUG:boto:Params: {} 2023-09-29T22:17:41.276 DEBUG:boto:Token: None 2023-09-29T22:17:41.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.277 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5a230ab1fedc0a-0065174d05-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.280 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.280 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone3, test-zone1} 2023-09-29T22:17:41.280 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone3, test-zone2} 2023-09-29T22:17:41.280 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.280 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.280 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.281 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:41.281 DEBUG:boto:Data: 2023-09-29T22:17:41.281 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.281 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.281 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.281 DEBUG:boto:Params: {} 2023-09-29T22:17:41.281 DEBUG:boto:Token: None 2023-09-29T22:17:41.281 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.281 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3NKPnVAPThkS4MdESi2f5JV66X8= 2023-09-29T22:17:41.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3NKPnVAPThkS4MdESi2f5JV66X8=', 'Content-Length': '0'} 2023-09-29T22:17:41.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f52c8d14ab16130b-0065174d05-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.284 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.284 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.284 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.284 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:41.284 DEBUG:boto:Data: 2023-09-29T22:17:41.285 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.285 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.285 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.285 DEBUG:boto:Params: {} 2023-09-29T22:17:41.285 DEBUG:boto:Token: None 2023-09-29T22:17:41.285 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.285 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3NKPnVAPThkS4MdESi2f5JV66X8= 2023-09-29T22:17:41.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3NKPnVAPThkS4MdESi2f5JV66X8=', 'Content-Length': '0'} 2023-09-29T22:17:41.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041af14cf94437c73-0065174d05-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:41.288 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.288 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.288 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.288 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.288 DEBUG:boto:Method: GET 2023-09-29T22:17:41.289 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.289 DEBUG:boto:Data: 2023-09-29T22:17:41.289 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.289 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.289 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.289 DEBUG:boto:Params: {} 2023-09-29T22:17:41.289 DEBUG:boto:Token: None 2023-09-29T22:17:41.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/?versions 2023-09-29T22:17:41.289 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c= 2023-09-29T22:17:41.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c=', 'Content-Length': '0'} 2023-09-29T22:17:41.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede1422a07fc6447-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.292 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.293 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:41.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:41.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:41.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:41.293 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:41.293 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.293 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.293 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.294 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.294 DEBUG:boto:Method: GET 2023-09-29T22:17:41.294 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.294 DEBUG:boto:Data: 2023-09-29T22:17:41.294 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.294 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.294 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.294 DEBUG:boto:Params: {} 2023-09-29T22:17:41.294 DEBUG:boto:Token: None 2023-09-29T22:17:41.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/?versions 2023-09-29T22:17:41.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c= 2023-09-29T22:17:41.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c=', 'Content-Length': '0'} 2023-09-29T22:17:41.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e901bcef3e32ab39-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.297 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.298 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:17:41.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:17:41.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:17:41.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:17:41.298 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.298 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.298 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.299 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.299 DEBUG:boto:Method: GET 2023-09-29T22:17:41.299 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.299 DEBUG:boto:Data: 2023-09-29T22:17:41.299 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.299 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.299 DEBUG:boto:Params: {} 2023-09-29T22:17:41.299 DEBUG:boto:Token: None 2023-09-29T22:17:41.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/?versions 2023-09-29T22:17:41.300 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c= 2023-09-29T22:17:41.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c=', 'Content-Length': '0'} 2023-09-29T22:17:41.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b11c608cb7aeb8d-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.302 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.303 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.303 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.303 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.303 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.303 DEBUG:boto:Method: GET 2023-09-29T22:17:41.303 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:17:41.303 DEBUG:boto:Data: 2023-09-29T22:17:41.303 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.304 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.304 DEBUG:boto:Params: {} 2023-09-29T22:17:41.304 DEBUG:boto:Token: None 2023-09-29T22:17:41.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/?versions 2023-09-29T22:17:41.304 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c= 2023-09-29T22:17:41.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wtVkHQM7jOSnhwIGOJ+a3CgaU/c=', 'Content-Length': '0'} 2023-09-29T22:17:41.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000363267dcf5b8d7c5-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.306 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:41.307 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.307 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.307 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.307 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.308 DEBUG:boto:Method: GET 2023-09-29T22:17:41.308 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.308 DEBUG:boto:Data: 2023-09-29T22:17:41.308 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.308 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.308 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.308 DEBUG:boto:Params: {} 2023-09-29T22:17:41.308 DEBUG:boto:Token: None 2023-09-29T22:17:41.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000067bca385843647bd-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.310 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.311 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.311 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.311 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.311 DEBUG:boto:Method: GET 2023-09-29T22:17:41.311 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.311 DEBUG:boto:Data: 2023-09-29T22:17:41.311 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.311 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.311 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.311 DEBUG:boto:Params: {} 2023-09-29T22:17:41.312 DEBUG:boto:Token: None 2023-09-29T22:17:41.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc26e55e4aa07700-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.314 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.314 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.314 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.314 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.315 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.315 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.315 DEBUG:boto:Data: 2023-09-29T22:17:41.315 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.315 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.315 DEBUG:boto:Params: {} 2023-09-29T22:17:41.315 DEBUG:boto:Token: None 2023-09-29T22:17:41.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.315 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PmULT3EbHCtOzShgsLazl0OHPT0= 2023-09-29T22:17:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PmULT3EbHCtOzShgsLazl0OHPT0=', 'Content-Length': '0'} 2023-09-29T22:17:41.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004fab122691be4af1-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.317 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.318 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.318 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.318 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.318 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.318 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.318 DEBUG:boto:Data: 2023-09-29T22:17:41.318 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.318 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.318 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.319 DEBUG:boto:Params: {} 2023-09-29T22:17:41.319 DEBUG:boto:Token: None 2023-09-29T22:17:41.319 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PmULT3EbHCtOzShgsLazl0OHPT0= 2023-09-29T22:17:41.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PmULT3EbHCtOzShgsLazl0OHPT0=', 'Content-Length': '0'} 2023-09-29T22:17:41.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003fdfecde0cf118a2-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:41.321 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.321 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.321 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.321 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.322 DEBUG:boto:Method: GET 2023-09-29T22:17:41.322 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.322 DEBUG:boto:Data: 2023-09-29T22:17:41.322 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.322 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.322 DEBUG:boto:Params: {} 2023-09-29T22:17:41.322 DEBUG:boto:Token: None 2023-09-29T22:17:41.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fa4759bcdbb4263-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.325 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.325 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.325 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.325 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.325 DEBUG:boto:Method: GET 2023-09-29T22:17:41.325 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.325 DEBUG:boto:Data: 2023-09-29T22:17:41.325 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.325 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.325 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.326 DEBUG:boto:Params: {} 2023-09-29T22:17:41.326 DEBUG:boto:Token: None 2023-09-29T22:17:41.326 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.326 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007451a26b65b7b521-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.328 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.328 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.328 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.328 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.328 DEBUG:boto:Method: GET 2023-09-29T22:17:41.329 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.329 DEBUG:boto:Data: 2023-09-29T22:17:41.329 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.329 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.329 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.329 DEBUG:boto:Params: {} 2023-09-29T22:17:41.329 DEBUG:boto:Token: None 2023-09-29T22:17:41.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.330 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ada2e0b88a2e7ab-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.378 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.379 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.379 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.379 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.379 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.379 DEBUG:boto:Method: GET 2023-09-29T22:17:41.379 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.380 DEBUG:boto:Data: 2023-09-29T22:17:41.380 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.380 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.380 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.380 DEBUG:boto:Params: {} 2023-09-29T22:17:41.380 DEBUG:boto:Token: None 2023-09-29T22:17:41.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cd4ed94ab42562-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.383 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.383 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.384 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.384 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.384 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:41.384 DEBUG:boto:Data: 2023-09-29T22:17:41.384 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.384 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.384 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.384 DEBUG:boto:Params: {} 2023-09-29T22:17:41.384 DEBUG:boto:Token: None 2023-09-29T22:17:41.385 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy 2023-09-29T22:17:41.385 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LiSifKz8qgnOKmYJeAzqx7lc/ks= 2023-09-29T22:17:41.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LiSifKz8qgnOKmYJeAzqx7lc/ks=', 'Content-Length': '0'} 2023-09-29T22:17:41.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000321e2ac5ec9c65f0-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.387 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.387 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.387 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.387 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:17:41.387 DEBUG:boto:Data: 2023-09-29T22:17:41.388 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.388 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.388 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.388 DEBUG:boto:Params: {} 2023-09-29T22:17:41.388 DEBUG:boto:Token: None 2023-09-29T22:17:41.388 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy 2023-09-29T22:17:41.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LiSifKz8qgnOKmYJeAzqx7lc/ks= 2023-09-29T22:17:41.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LiSifKz8qgnOKmYJeAzqx7lc/ks=', 'Content-Length': '0'} 2023-09-29T22:17:41.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003489d589b81e1114-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:17:41.391 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.391 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.391 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.391 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.391 DEBUG:boto:Method: GET 2023-09-29T22:17:41.391 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.392 DEBUG:boto:Data: 2023-09-29T22:17:41.392 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.392 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.392 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.392 DEBUG:boto:Params: {} 2023-09-29T22:17:41.392 DEBUG:boto:Token: None 2023-09-29T22:17:41.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000591ecebe06b45eee-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.395 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:17:41.395 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:17:41.395 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.395 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.395 DEBUG:boto:Method: GET 2023-09-29T22:17:41.395 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.395 DEBUG:boto:Data: 2023-09-29T22:17:41.395 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.395 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.395 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.396 DEBUG:boto:Params: {} 2023-09-29T22:17:41.396 DEBUG:boto:Token: None 2023-09-29T22:17:41.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:17:41.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw= 2023-09-29T22:17:41.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O9h8Jc+RBdEi3XT2ARpPn/A0Sqw=', 'Content-Length': '0'} 2023-09-29T22:17:41.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e742b837fdaa876-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.398 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.398 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.398 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.398 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.399 DEBUG:boto:Method: GET 2023-09-29T22:17:41.399 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.399 DEBUG:boto:Data: 2023-09-29T22:17:41.399 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.399 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.399 DEBUG:boto:Params: {} 2023-09-29T22:17:41.399 DEBUG:boto:Token: None 2023-09-29T22:17:41.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee11d908733bdc4a-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.449 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.451 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.451 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.451 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.451 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.451 DEBUG:boto:Method: GET 2023-09-29T22:17:41.451 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:17:41.451 DEBUG:boto:Data: 2023-09-29T22:17:41.451 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.451 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.451 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.452 DEBUG:boto:Params: {} 2023-09-29T22:17:41.452 DEBUG:boto:Token: None 2023-09-29T22:17:41.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7c34df88f805d3-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.455 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:41.455 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.455 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.456 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.456 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.456 DEBUG:boto:Method: GET 2023-09-29T22:17:41.456 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.456 DEBUG:boto:Data: 2023-09-29T22:17:41.456 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.456 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.456 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.457 DEBUG:boto:Params: {} 2023-09-29T22:17:41.457 DEBUG:boto:Token: None 2023-09-29T22:17:41.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.457 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fd71c0e078a4c79-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.459 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.459 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.459 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.459 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.460 DEBUG:boto:Method: GET 2023-09-29T22:17:41.460 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.460 DEBUG:boto:Data: 2023-09-29T22:17:41.460 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.460 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.460 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.460 DEBUG:boto:Params: {} 2023-09-29T22:17:41.460 DEBUG:boto:Token: None 2023-09-29T22:17:41.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025021785c4a4c658-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.506 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.507 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.507 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.507 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.507 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.507 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.507 DEBUG:boto:Data: 2023-09-29T22:17:41.507 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.507 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.508 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.508 DEBUG:boto:Params: {} 2023-09-29T22:17:41.508 DEBUG:boto:Token: None 2023-09-29T22:17:41.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtp3h0SEw6tD7s2mcpKouTGNnYk= 2023-09-29T22:17:41.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtp3h0SEw6tD7s2mcpKouTGNnYk=', 'Content-Length': '0'} 2023-09-29T22:17:41.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e46dcd2f37c01907-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.510 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.510 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.510 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.511 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.511 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.511 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.511 DEBUG:boto:Data: 2023-09-29T22:17:41.511 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.511 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.511 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.511 DEBUG:boto:Params: {} 2023-09-29T22:17:41.512 DEBUG:boto:Token: None 2023-09-29T22:17:41.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtp3h0SEw6tD7s2mcpKouTGNnYk= 2023-09-29T22:17:41.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtp3h0SEw6tD7s2mcpKouTGNnYk=', 'Content-Length': '0'} 2023-09-29T22:17:41.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061fddd133b844d3f-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:41.514 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.514 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.514 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.514 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.515 DEBUG:boto:Method: GET 2023-09-29T22:17:41.515 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.515 DEBUG:boto:Data: 2023-09-29T22:17:41.515 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.515 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.515 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.515 DEBUG:boto:Params: {} 2023-09-29T22:17:41.515 DEBUG:boto:Token: None 2023-09-29T22:17:41.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008f0a29a38c41955-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.562 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.563 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.563 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.563 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.563 DEBUG:boto:Method: GET 2023-09-29T22:17:41.563 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.563 DEBUG:boto:Data: 2023-09-29T22:17:41.563 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.563 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.563 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.564 DEBUG:boto:Params: {} 2023-09-29T22:17:41.564 DEBUG:boto:Token: None 2023-09-29T22:17:41.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df5c8230675fe91e-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.614 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.615 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.615 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.615 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.615 DEBUG:boto:Method: GET 2023-09-29T22:17:41.615 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.615 DEBUG:boto:Data: 2023-09-29T22:17:41.615 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.615 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.616 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.616 DEBUG:boto:Params: {} 2023-09-29T22:17:41.616 DEBUG:boto:Token: None 2023-09-29T22:17:41.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f2073e9fab562f-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.619 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.619 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.619 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.620 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.620 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.620 DEBUG:boto:Method: GET 2023-09-29T22:17:41.620 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.620 DEBUG:boto:Data: 2023-09-29T22:17:41.620 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.620 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.620 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.620 DEBUG:boto:Params: {} 2023-09-29T22:17:41.620 DEBUG:boto:Token: None 2023-09-29T22:17:41.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.621 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca2dc5f9dd932c6-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.666 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.667 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.667 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.667 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.667 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.667 DEBUG:boto:Data: 2023-09-29T22:17:41.668 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.668 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.668 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.668 DEBUG:boto:Params: {} 2023-09-29T22:17:41.668 DEBUG:boto:Token: None 2023-09-29T22:17:41.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/DhRkQTX0NOFVvyrG+r2L/07t2U= 2023-09-29T22:17:41.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/DhRkQTX0NOFVvyrG+r2L/07t2U=', 'Content-Length': '0'} 2023-09-29T22:17:41.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001697479df160e67e-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.672 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.672 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.672 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.672 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.672 DEBUG:boto:Data: 2023-09-29T22:17:41.672 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.672 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.672 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.672 DEBUG:boto:Params: {} 2023-09-29T22:17:41.673 DEBUG:boto:Token: None 2023-09-29T22:17:41.673 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.673 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/DhRkQTX0NOFVvyrG+r2L/07t2U= 2023-09-29T22:17:41.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/DhRkQTX0NOFVvyrG+r2L/07t2U=', 'Content-Length': '0'} 2023-09-29T22:17:41.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002421a46ea8debbcc-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:17:41.676 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.676 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.676 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.676 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.677 DEBUG:boto:Method: GET 2023-09-29T22:17:41.677 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.677 DEBUG:boto:Data: 2023-09-29T22:17:41.677 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.677 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.677 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.678 DEBUG:boto:Params: {} 2023-09-29T22:17:41.678 DEBUG:boto:Token: None 2023-09-29T22:17:41.678 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.678 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b06867026aecb64-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.722 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.723 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:41.723 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.723 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.723 DEBUG:boto:Method: GET 2023-09-29T22:17:41.723 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.723 DEBUG:boto:Data: 2023-09-29T22:17:41.723 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.723 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.723 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.724 DEBUG:boto:Params: {} 2023-09-29T22:17:41.724 DEBUG:boto:Token: None 2023-09-29T22:17:41.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:17:41.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I= 2023-09-29T22:17:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AIbW2kSkZssLFsJjwls2xzcPd+I=', 'Content-Length': '0'} 2023-09-29T22:17:41.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000911132eabc095946-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.770 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.771 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.771 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.771 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.771 DEBUG:boto:Method: GET 2023-09-29T22:17:41.771 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.771 DEBUG:boto:Data: 2023-09-29T22:17:41.771 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.771 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.772 DEBUG:boto:Params: {} 2023-09-29T22:17:41.772 DEBUG:boto:Token: None 2023-09-29T22:17:41.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a122ccd762fd0f2-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.775 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.775 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.775 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.775 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.776 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.776 DEBUG:boto:Method: GET 2023-09-29T22:17:41.776 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:17:41.776 DEBUG:boto:Data: 2023-09-29T22:17:41.776 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.776 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.776 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.776 DEBUG:boto:Params: {} 2023-09-29T22:17:41.776 DEBUG:boto:Token: None 2023-09-29T22:17:41.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.777 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33d9d6e9762769b-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.822 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:41.823 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.823 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.823 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.823 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.823 DEBUG:boto:Method: GET 2023-09-29T22:17:41.824 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.824 DEBUG:boto:Data: 2023-09-29T22:17:41.824 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.824 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.824 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.824 DEBUG:boto:Params: {} 2023-09-29T22:17:41.824 DEBUG:boto:Token: None 2023-09-29T22:17:41.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.825 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU= 2023-09-29T22:17:41.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU=', 'Content-Length': '0'} 2023-09-29T22:17:41.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000519c98bf396f3e84-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.827 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.827 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.827 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.827 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.828 DEBUG:boto:Method: GET 2023-09-29T22:17:41.828 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.828 DEBUG:boto:Data: 2023-09-29T22:17:41.828 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.828 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.828 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.828 DEBUG:boto:Params: {} 2023-09-29T22:17:41.828 DEBUG:boto:Token: None 2023-09-29T22:17:41.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU= 2023-09-29T22:17:41.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU=', 'Content-Length': '0'} 2023-09-29T22:17:41.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071bc98636280b42b-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.830 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.831 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.831 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.831 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.831 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.831 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.831 DEBUG:boto:Data: 2023-09-29T22:17:41.831 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.831 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.831 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.832 DEBUG:boto:Params: {} 2023-09-29T22:17:41.832 DEBUG:boto:Token: None 2023-09-29T22:17:41.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1JHC0rW3HTYEdkXYu0dop2zqnfA= 2023-09-29T22:17:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1JHC0rW3HTYEdkXYu0dop2zqnfA=', 'Content-Length': '0'} 2023-09-29T22:17:41.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d142b1519e08222-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.834 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.834 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.834 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.835 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.835 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.835 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.835 DEBUG:boto:Data: 2023-09-29T22:17:41.835 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.835 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.835 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.835 DEBUG:boto:Params: {} 2023-09-29T22:17:41.835 DEBUG:boto:Token: None 2023-09-29T22:17:41.836 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1JHC0rW3HTYEdkXYu0dop2zqnfA= 2023-09-29T22:17:41.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1JHC0rW3HTYEdkXYu0dop2zqnfA=', 'Content-Length': '0'} 2023-09-29T22:17:41.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008035184fbacc0487-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:41.838 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.838 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.838 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.838 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.838 DEBUG:boto:Method: GET 2023-09-29T22:17:41.838 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.839 DEBUG:boto:Data: 2023-09-29T22:17:41.839 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.839 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.839 DEBUG:boto:Params: {} 2023-09-29T22:17:41.839 DEBUG:boto:Token: None 2023-09-29T22:17:41.839 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.839 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU= 2023-09-29T22:17:41.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU=', 'Content-Length': '0'} 2023-09-29T22:17:41.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6a0befa8c353305-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.886 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.886 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.887 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.887 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.887 DEBUG:boto:Method: GET 2023-09-29T22:17:41.887 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.887 DEBUG:boto:Data: 2023-09-29T22:17:41.887 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.887 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.887 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.887 DEBUG:boto:Params: {} 2023-09-29T22:17:41.887 DEBUG:boto:Token: None 2023-09-29T22:17:41.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU= 2023-09-29T22:17:41.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU=', 'Content-Length': '0'} 2023-09-29T22:17:41.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071306e61007d4762-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.891 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.891 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.891 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.891 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.891 DEBUG:boto:Method: GET 2023-09-29T22:17:41.891 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.891 DEBUG:boto:Data: 2023-09-29T22:17:41.891 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.891 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.892 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.892 DEBUG:boto:Params: {} 2023-09-29T22:17:41.892 DEBUG:boto:Token: None 2023-09-29T22:17:41.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156208c5d8d432d1-0065174d05-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.938 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.939 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:41.939 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:41.939 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.939 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.939 DEBUG:boto:Method: GET 2023-09-29T22:17:41.939 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:41.939 DEBUG:boto:Data: 2023-09-29T22:17:41.939 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.940 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.940 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.940 DEBUG:boto:Params: {} 2023-09-29T22:17:41.940 DEBUG:boto:Token: None 2023-09-29T22:17:41.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/ 2023-09-29T22:17:41.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo= 2023-09-29T22:17:41.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JIC6g0hlqD1euZzBDR0XAL3E4Uo=', 'Content-Length': '0'} 2023-09-29T22:17:41.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1953dd333c99b55-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.943 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:41.943 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.943 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.944 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.944 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.944 DEBUG:boto:Data: 2023-09-29T22:17:41.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.944 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.944 DEBUG:boto:Params: {} 2023-09-29T22:17:41.944 DEBUG:boto:Token: None 2023-09-29T22:17:41.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YeZDVLymFoYkq78wygBB6OggsS8= 2023-09-29T22:17:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YeZDVLymFoYkq78wygBB6OggsS8=', 'Content-Length': '0'} 2023-09-29T22:17:41.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b4ea11c212c8566-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.947 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.947 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.948 DEBUG:boto:Method: HEAD 2023-09-29T22:17:41.948 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.948 DEBUG:boto:Data: 2023-09-29T22:17:41.948 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.948 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.948 DEBUG:boto:Port: 8000 2023-09-29T22:17:41.948 DEBUG:boto:Params: {} 2023-09-29T22:17:41.948 DEBUG:boto:Token: None 2023-09-29T22:17:41.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YeZDVLymFoYkq78wygBB6OggsS8= 2023-09-29T22:17:41.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YeZDVLymFoYkq78wygBB6OggsS8=', 'Content-Length': '0'} 2023-09-29T22:17:41.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b6c8d7319c472ac-0065174d05-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:17:41.951 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.951 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.951 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.951 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.951 DEBUG:boto:Method: GET 2023-09-29T22:17:41.952 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.952 DEBUG:boto:Data: 2023-09-29T22:17:41.952 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.952 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:41.952 DEBUG:boto:Port: 8001 2023-09-29T22:17:41.952 DEBUG:boto:Params: {} 2023-09-29T22:17:41.952 DEBUG:boto:Token: None 2023-09-29T22:17:41.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:41 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU= 2023-09-29T22:17:41.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4vVKYNUDk6lEmWJFqO/xubhYksU=', 'Content-Length': '0'} 2023-09-29T22:17:41.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce6f561725400be2-0065174d05-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:41.998 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.999 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:41.999 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.999 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.999 DEBUG:boto:Method: GET 2023-09-29T22:17:41.999 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:41.999 DEBUG:boto:Data: 2023-09-29T22:17:41.999 DEBUG:boto:Headers: {} 2023-09-29T22:17:41.999 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:41.999 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.000 DEBUG:boto:Params: {} 2023-09-29T22:17:42.000 DEBUG:boto:Token: None 2023-09-29T22:17:42.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:17:42.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w5v6jyvXNr68SD3YFFN09/sgNlE= 2023-09-29T22:17:42.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w5v6jyvXNr68SD3YFFN09/sgNlE=', 'Content-Length': '0'} 2023-09-29T22:17:42.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa90c27a0160786a-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.003 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.003 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.003 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.003 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.003 DEBUG:boto:Method: GET 2023-09-29T22:17:42.003 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.003 DEBUG:boto:Data: 2023-09-29T22:17:42.003 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.003 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.003 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.004 DEBUG:boto:Params: {} 2023-09-29T22:17:42.004 DEBUG:boto:Token: None 2023-09-29T22:17:42.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004501286d0a06c3e2-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.054 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.055 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.055 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.055 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.055 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.055 DEBUG:boto:Method: GET 2023-09-29T22:17:42.055 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:17:42.055 DEBUG:boto:Data: 2023-09-29T22:17:42.056 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.056 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.056 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.056 DEBUG:boto:Params: {} 2023-09-29T22:17:42.056 DEBUG:boto:Token: None 2023-09-29T22:17:42.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf41d19df781935-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.059 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:42.060 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.060 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.060 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.060 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.060 DEBUG:boto:Method: GET 2023-09-29T22:17:42.060 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.060 DEBUG:boto:Data: 2023-09-29T22:17:42.061 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.061 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.061 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.061 DEBUG:boto:Params: {} 2023-09-29T22:17:42.061 DEBUG:boto:Token: None 2023-09-29T22:17:42.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.062 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f239a28982fa792-0065174d06-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.066 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.066 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.066 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.066 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.067 DEBUG:boto:Method: GET 2023-09-29T22:17:42.067 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.067 DEBUG:boto:Data: 2023-09-29T22:17:42.067 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.067 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.067 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.067 DEBUG:boto:Params: {} 2023-09-29T22:17:42.067 DEBUG:boto:Token: None 2023-09-29T22:17:42.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f274e86be8d40a2-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.070 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.070 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.071 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.071 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.071 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.071 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.071 DEBUG:boto:Data: 2023-09-29T22:17:42.071 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.071 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.071 DEBUG:boto:Params: {} 2023-09-29T22:17:42.072 DEBUG:boto:Token: None 2023-09-29T22:17:42.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:whq3KSAiUUDfbBc4qEeM2c5KMSs= 2023-09-29T22:17:42.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:whq3KSAiUUDfbBc4qEeM2c5KMSs=', 'Content-Length': '0'} 2023-09-29T22:17:42.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e905a8f3411af065-0065174d06-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.074 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.074 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.074 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.074 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.074 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.075 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.075 DEBUG:boto:Data: 2023-09-29T22:17:42.075 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.075 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.075 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.075 DEBUG:boto:Params: {} 2023-09-29T22:17:42.075 DEBUG:boto:Token: None 2023-09-29T22:17:42.075 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:whq3KSAiUUDfbBc4qEeM2c5KMSs= 2023-09-29T22:17:42.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:whq3KSAiUUDfbBc4qEeM2c5KMSs=', 'Content-Length': '0'} 2023-09-29T22:17:42.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b7ef7b46b755587-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:42.078 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.078 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.078 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.078 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.079 DEBUG:boto:Method: GET 2023-09-29T22:17:42.079 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.079 DEBUG:boto:Data: 2023-09-29T22:17:42.079 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.079 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.079 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.079 DEBUG:boto:Params: {} 2023-09-29T22:17:42.079 DEBUG:boto:Token: None 2023-09-29T22:17:42.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.080 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e229c656c0a8cfe-0065174d06-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.082 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.082 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.082 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.083 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.083 DEBUG:boto:Method: GET 2023-09-29T22:17:42.083 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.083 DEBUG:boto:Data: 2023-09-29T22:17:42.083 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.083 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.083 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.083 DEBUG:boto:Params: {} 2023-09-29T22:17:42.084 DEBUG:boto:Token: None 2023-09-29T22:17:42.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db5127d3ecda70bb-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.126 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.127 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.127 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.127 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.127 DEBUG:boto:Method: GET 2023-09-29T22:17:42.127 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.127 DEBUG:boto:Data: 2023-09-29T22:17:42.128 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.128 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.128 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.128 DEBUG:boto:Params: {} 2023-09-29T22:17:42.128 DEBUG:boto:Token: None 2023-09-29T22:17:42.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.129 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe0cd8620285343-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.131 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.132 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.132 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.132 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.132 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.132 DEBUG:boto:Method: GET 2023-09-29T22:17:42.132 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.132 DEBUG:boto:Data: 2023-09-29T22:17:42.132 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.133 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.133 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.133 DEBUG:boto:Params: {} 2023-09-29T22:17:42.133 DEBUG:boto:Token: None 2023-09-29T22:17:42.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.133 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d49b0a6ccb33706-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.136 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.136 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.136 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.136 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.137 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.137 DEBUG:boto:Data: 2023-09-29T22:17:42.137 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.137 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.137 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.137 DEBUG:boto:Params: {} 2023-09-29T22:17:42.137 DEBUG:boto:Token: None 2023-09-29T22:17:42.137 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ppgkhe3C06LP83jfAZP0CFaKE1Y= 2023-09-29T22:17:42.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ppgkhe3C06LP83jfAZP0CFaKE1Y=', 'Content-Length': '0'} 2023-09-29T22:17:42.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003ac7429cf4b15e9-0065174d06-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.140 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.140 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.141 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.141 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.141 DEBUG:boto:Data: 2023-09-29T22:17:42.141 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.141 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.141 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.141 DEBUG:boto:Params: {} 2023-09-29T22:17:42.141 DEBUG:boto:Token: None 2023-09-29T22:17:42.142 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.142 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ppgkhe3C06LP83jfAZP0CFaKE1Y= 2023-09-29T22:17:42.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ppgkhe3C06LP83jfAZP0CFaKE1Y=', 'Content-Length': '0'} 2023-09-29T22:17:42.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000791309423d3f7928-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:17:42.144 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.144 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.145 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.145 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.145 DEBUG:boto:Method: GET 2023-09-29T22:17:42.145 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.145 DEBUG:boto:Data: 2023-09-29T22:17:42.145 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.145 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.145 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.145 DEBUG:boto:Params: {} 2023-09-29T22:17:42.146 DEBUG:boto:Token: None 2023-09-29T22:17:42.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.146 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022573da991c63604-0065174d06-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.148 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.149 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.149 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.149 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.149 DEBUG:boto:Method: GET 2023-09-29T22:17:42.149 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.149 DEBUG:boto:Data: 2023-09-29T22:17:42.149 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.149 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.149 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.149 DEBUG:boto:Params: {} 2023-09-29T22:17:42.150 DEBUG:boto:Token: None 2023-09-29T22:17:42.150 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:17:42.150 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M= 2023-09-29T22:17:42.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QPMQIu8zRJC7hfQZJ9Z87osoN/M=', 'Content-Length': '0'} 2023-09-29T22:17:42.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8ae86e7b17e81b0-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.198 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.198 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.199 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.199 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.199 DEBUG:boto:Method: GET 2023-09-29T22:17:42.199 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.199 DEBUG:boto:Data: 2023-09-29T22:17:42.199 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.199 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.199 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.199 DEBUG:boto:Params: {} 2023-09-29T22:17:42.199 DEBUG:boto:Token: None 2023-09-29T22:17:42.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829f6f2bd7fbb699-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.202 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.203 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.203 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.203 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.203 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.203 DEBUG:boto:Method: GET 2023-09-29T22:17:42.203 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:17:42.203 DEBUG:boto:Data: 2023-09-29T22:17:42.204 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.204 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.204 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.204 DEBUG:boto:Params: {} 2023-09-29T22:17:42.204 DEBUG:boto:Token: None 2023-09-29T22:17:42.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs= 2023-09-29T22:17:42.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1NQ6F0NHNIlwzurDVBf76TsOpcs=', 'Content-Length': '0'} 2023-09-29T22:17:42.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000363c35ce7cf0f41e-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.207 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.207 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone3, test-zone2} 2023-09-29T22:17:42.207 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.207 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.207 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.208 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:42.208 DEBUG:boto:Data: 2023-09-29T22:17:42.208 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.208 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.208 DEBUG:boto:Params: {} 2023-09-29T22:17:42.208 DEBUG:boto:Token: None 2023-09-29T22:17:42.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/ 2023-09-29T22:17:42.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA= 2023-09-29T22:17:42.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA=', 'Content-Length': '0'} 2023-09-29T22:17:42.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000095aff1144d3b928-0065174d06-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.211 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.211 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.211 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.211 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.212 DEBUG:boto:Method: GET 2023-09-29T22:17:42.212 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.212 DEBUG:boto:Data: 2023-09-29T22:17:42.212 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.212 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.212 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.212 DEBUG:boto:Params: {} 2023-09-29T22:17:42.212 DEBUG:boto:Token: None 2023-09-29T22:17:42.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/?versions 2023-09-29T22:17:42.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw= 2023-09-29T22:17:42.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw=', 'Content-Length': '0'} 2023-09-29T22:17:42.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8bb777f54599408-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.258 DEBUG:boto:b'dmuhsu-16obj-test-zone1999falseobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.259 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:42.259 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:17:42.259 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?acl 2023-09-29T22:17:42.259 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?acl 2023-09-29T22:17:42.259 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.259 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?acl 2023-09-29T22:17:42.259 DEBUG:boto:Data: 2023-09-29T22:17:42.259 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.259 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.260 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.260 DEBUG:boto:Params: {} 2023-09-29T22:17:42.260 DEBUG:boto:Token: None 2023-09-29T22:17:42.260 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-16/obj-test-zone1?acl 2023-09-29T22:17:42.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pPB224h4TLGc6TH039ogWHRVrlA= 2023-09-29T22:17:42.260 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pPB224h4TLGc6TH039ogWHRVrlA=', 'Content-Length': '0'} 2023-09-29T22:17:42.267 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ea09a35695623cb3-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.267 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.267 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.267 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.268 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:42.268 DEBUG:boto:Data: 2023-09-29T22:17:42.268 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.268 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.268 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.268 DEBUG:boto:Params: {} 2023-09-29T22:17:42.268 DEBUG:boto:Token: None 2023-09-29T22:17:42.268 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/ 2023-09-29T22:17:42.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA= 2023-09-29T22:17:42.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA=', 'Content-Length': '0'} 2023-09-29T22:17:42.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3f12ea14e97997b-0065174d06-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.273 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.273 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.273 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.274 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.274 DEBUG:boto:Method: GET 2023-09-29T22:17:42.274 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.274 DEBUG:boto:Data: 2023-09-29T22:17:42.274 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.274 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.274 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.274 DEBUG:boto:Params: {} 2023-09-29T22:17:42.274 DEBUG:boto:Token: None 2023-09-29T22:17:42.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/?versions 2023-09-29T22:17:42.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw= 2023-09-29T22:17:42.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw=', 'Content-Length': '0'} 2023-09-29T22:17:42.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d9d4330690ce40-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.322 DEBUG:boto:b'dmuhsu-16obj-test-zone2999falseobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.323 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:42.323 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:17:42.323 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?acl 2023-09-29T22:17:42.323 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?acl 2023-09-29T22:17:42.323 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.323 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?acl 2023-09-29T22:17:42.323 DEBUG:boto:Data: 2023-09-29T22:17:42.324 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.324 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.324 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.324 DEBUG:boto:Params: {} 2023-09-29T22:17:42.324 DEBUG:boto:Token: None 2023-09-29T22:17:42.324 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-16/obj-test-zone2?acl 2023-09-29T22:17:42.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gjA1+696INgQXXAXO2zxiramHTQ= 2023-09-29T22:17:42.324 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gjA1+696INgQXXAXO2zxiramHTQ=', 'Content-Length': '0'} 2023-09-29T22:17:42.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c51870387a988dc-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.332 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.332 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.332 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.332 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:17:42.332 DEBUG:boto:Data: 2023-09-29T22:17:42.332 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.333 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.333 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.333 DEBUG:boto:Params: {} 2023-09-29T22:17:42.333 DEBUG:boto:Token: None 2023-09-29T22:17:42.333 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/ 2023-09-29T22:17:42.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA= 2023-09-29T22:17:42.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IKrT2gm053EzGjzVLKtMxuX47zA=', 'Content-Length': '0'} 2023-09-29T22:17:42.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000079b2d659b16594a-0065174d06-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.336 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:17:42.336 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:17:42.336 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.336 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.336 DEBUG:boto:Method: GET 2023-09-29T22:17:42.336 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.336 DEBUG:boto:Data: 2023-09-29T22:17:42.337 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.337 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.337 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.337 DEBUG:boto:Params: {} 2023-09-29T22:17:42.337 DEBUG:boto:Token: None 2023-09-29T22:17:42.337 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-16/?versions 2023-09-29T22:17:42.337 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw= 2023-09-29T22:17:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x++zqPolo4xMscl0EhwOklJJnKw=', 'Content-Length': '0'} 2023-09-29T22:17:42.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b1b1320909e7c87-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.386 DEBUG:boto:b'dmuhsu-16obj-test-zone3999falseobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.387 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:42.387 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:17:42.387 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?acl 2023-09-29T22:17:42.387 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?acl 2023-09-29T22:17:42.387 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.387 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?acl 2023-09-29T22:17:42.387 DEBUG:boto:Data: 2023-09-29T22:17:42.388 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.388 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.388 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.388 DEBUG:boto:Params: {} 2023-09-29T22:17:42.388 DEBUG:boto:Token: None 2023-09-29T22:17:42.388 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-16/obj-test-zone3?acl 2023-09-29T22:17:42.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:960laMmab/Y12oXd9j0KeiXh3xg= 2023-09-29T22:17:42.388 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:960laMmab/Y12oXd9j0KeiXh3xg=', 'Content-Length': '0'} 2023-09-29T22:17:42.395 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c280d38b25809f2d-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.395 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.395 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.395 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.395 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:42.396 DEBUG:boto:Data: 2023-09-29T22:17:42.396 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.396 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.396 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.396 DEBUG:boto:Params: {} 2023-09-29T22:17:42.396 DEBUG:boto:Token: None 2023-09-29T22:17:42.396 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/ 2023-09-29T22:17:42.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA= 2023-09-29T22:17:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA=', 'Content-Length': '0'} 2023-09-29T22:17:42.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033d0534f94da7bec-0065174d06-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.399 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.399 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.399 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.399 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.399 DEBUG:boto:Method: GET 2023-09-29T22:17:42.399 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.399 DEBUG:boto:Data: 2023-09-29T22:17:42.399 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.400 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.400 DEBUG:boto:Params: {} 2023-09-29T22:17:42.400 DEBUG:boto:Token: None 2023-09-29T22:17:42.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/?versions 2023-09-29T22:17:42.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0= 2023-09-29T22:17:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0=', 'Content-Length': '0'} 2023-09-29T22:17:42.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe1dc62ab127b19-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.403 DEBUG:boto:b'dmuhsu-17obj-test-zone1999falseobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.403 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:42.403 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:17:42.403 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?acl 2023-09-29T22:17:42.403 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?acl 2023-09-29T22:17:42.403 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.403 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?acl 2023-09-29T22:17:42.404 DEBUG:boto:Data: 2023-09-29T22:17:42.404 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.404 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.404 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.404 DEBUG:boto:Params: {} 2023-09-29T22:17:42.404 DEBUG:boto:Token: None 2023-09-29T22:17:42.404 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-17/obj-test-zone1?acl 2023-09-29T22:17:42.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rwrQ/0oQwsuDq+sw/C2/4g1npIA= 2023-09-29T22:17:42.404 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rwrQ/0oQwsuDq+sw/C2/4g1npIA=', 'Content-Length': '0'} 2023-09-29T22:17:42.410 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077a2c047dd722fc5-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.410 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.411 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.411 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.411 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:42.411 DEBUG:boto:Data: 2023-09-29T22:17:42.411 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.411 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.411 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.411 DEBUG:boto:Params: {} 2023-09-29T22:17:42.411 DEBUG:boto:Token: None 2023-09-29T22:17:42.412 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/ 2023-09-29T22:17:42.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA= 2023-09-29T22:17:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA=', 'Content-Length': '0'} 2023-09-29T22:17:42.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000452ee69eff935485-0065174d06-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.414 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.415 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.415 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.415 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.415 DEBUG:boto:Method: GET 2023-09-29T22:17:42.415 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.415 DEBUG:boto:Data: 2023-09-29T22:17:42.415 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.415 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.416 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.416 DEBUG:boto:Params: {} 2023-09-29T22:17:42.416 DEBUG:boto:Token: None 2023-09-29T22:17:42.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/?versions 2023-09-29T22:17:42.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0= 2023-09-29T22:17:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0=', 'Content-Length': '0'} 2023-09-29T22:17:42.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae548716a5925002-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.418 DEBUG:boto:b'dmuhsu-17obj-test-zone2999falseobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.419 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:42.419 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:17:42.419 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?acl 2023-09-29T22:17:42.419 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?acl 2023-09-29T22:17:42.419 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.419 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?acl 2023-09-29T22:17:42.419 DEBUG:boto:Data: 2023-09-29T22:17:42.419 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.419 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.419 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.419 DEBUG:boto:Params: {} 2023-09-29T22:17:42.420 DEBUG:boto:Token: None 2023-09-29T22:17:42.420 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-17/obj-test-zone2?acl 2023-09-29T22:17:42.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uwdilclmcNqukClNW2N00jpRUz8= 2023-09-29T22:17:42.420 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uwdilclmcNqukClNW2N00jpRUz8=', 'Content-Length': '0'} 2023-09-29T22:17:42.427 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044453b5ba9644792-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.427 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.427 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.427 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.427 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:17:42.427 DEBUG:boto:Data: 2023-09-29T22:17:42.427 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.428 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.428 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.428 DEBUG:boto:Params: {} 2023-09-29T22:17:42.428 DEBUG:boto:Token: None 2023-09-29T22:17:42.428 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/ 2023-09-29T22:17:42.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA= 2023-09-29T22:17:42.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P7230N3/ny5zI4XfIbdGoKVvUTA=', 'Content-Length': '0'} 2023-09-29T22:17:42.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c2a90242c111910-0065174d06-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.431 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:17:42.431 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:17:42.431 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.431 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.431 DEBUG:boto:Method: GET 2023-09-29T22:17:42.431 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.432 DEBUG:boto:Data: 2023-09-29T22:17:42.432 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.432 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.432 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.432 DEBUG:boto:Params: {} 2023-09-29T22:17:42.432 DEBUG:boto:Token: None 2023-09-29T22:17:42.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-17/?versions 2023-09-29T22:17:42.433 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0= 2023-09-29T22:17:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a6jLzbWqNk2zAFgFbatZps5QQP0=', 'Content-Length': '0'} 2023-09-29T22:17:42.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345778cbe41a07f4-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.482 DEBUG:boto:b'dmuhsu-17obj-test-zone3999falseobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.483 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:42.483 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:17:42.483 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?acl 2023-09-29T22:17:42.483 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?acl 2023-09-29T22:17:42.483 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.483 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?acl 2023-09-29T22:17:42.483 DEBUG:boto:Data: 2023-09-29T22:17:42.483 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.484 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.484 DEBUG:boto:Params: {} 2023-09-29T22:17:42.484 DEBUG:boto:Token: None 2023-09-29T22:17:42.484 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-17/obj-test-zone3?acl 2023-09-29T22:17:42.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZlpaSR6PxWp2yFhYwrg9Ohr9770= 2023-09-29T22:17:42.484 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZlpaSR6PxWp2yFhYwrg9Ohr9770=', 'Content-Length': '0'} 2023-09-29T22:17:42.491 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047f04025bc470bb9-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.491 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.491 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.491 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.491 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:42.491 DEBUG:boto:Data: 2023-09-29T22:17:42.491 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.492 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.492 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.492 DEBUG:boto:Params: {} 2023-09-29T22:17:42.492 DEBUG:boto:Token: None 2023-09-29T22:17:42.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU= 2023-09-29T22:17:42.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU=', 'Content-Length': '0'} 2023-09-29T22:17:42.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d803645cad5a07f2-0065174d06-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.495 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.495 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.495 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.495 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.495 DEBUG:boto:Method: GET 2023-09-29T22:17:42.495 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone1 2023-09-29T22:17:42.495 DEBUG:boto:Data: 2023-09-29T22:17:42.495 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.495 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.496 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.496 DEBUG:boto:Params: {} 2023-09-29T22:17:42.496 DEBUG:boto:Token: None 2023-09-29T22:17:42.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/?versions 2023-09-29T22:17:42.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY= 2023-09-29T22:17:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY=', 'Content-Length': '0'} 2023-09-29T22:17:42.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045bdfdc5e1c8b9b8-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.499 DEBUG:boto:b'dmuhsu-18obj-test-zone1999falseobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.499 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:42.499 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:17:42.499 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?acl 2023-09-29T22:17:42.499 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?acl 2023-09-29T22:17:42.499 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.499 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?acl 2023-09-29T22:17:42.500 DEBUG:boto:Data: 2023-09-29T22:17:42.500 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.500 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:17:42.500 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.500 DEBUG:boto:Params: {} 2023-09-29T22:17:42.500 DEBUG:boto:Token: None 2023-09-29T22:17:42.500 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-18/obj-test-zone1?acl 2023-09-29T22:17:42.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KgS3ABS1XImJ+LCbPgWKxczcAE0= 2023-09-29T22:17:42.500 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KgS3ABS1XImJ+LCbPgWKxczcAE0=', 'Content-Length': '0'} 2023-09-29T22:17:42.507 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bbfa09ff497a28b-0065174d06-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.508 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.508 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.508 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.508 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:42.508 DEBUG:boto:Data: 2023-09-29T22:17:42.508 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.508 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.508 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.508 DEBUG:boto:Params: {} 2023-09-29T22:17:42.509 DEBUG:boto:Token: None 2023-09-29T22:17:42.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU= 2023-09-29T22:17:42.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU=', 'Content-Length': '0'} 2023-09-29T22:17:42.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645e5ec04dc552b1-0065174d06-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.511 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.511 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.511 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.512 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.512 DEBUG:boto:Method: GET 2023-09-29T22:17:42.512 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone2 2023-09-29T22:17:42.512 DEBUG:boto:Data: 2023-09-29T22:17:42.512 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.512 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.512 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.512 DEBUG:boto:Params: {} 2023-09-29T22:17:42.512 DEBUG:boto:Token: None 2023-09-29T22:17:42.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/?versions 2023-09-29T22:17:42.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY= 2023-09-29T22:17:42.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY=', 'Content-Length': '0'} 2023-09-29T22:17:42.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ac279df40c111b-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.515 DEBUG:boto:b'dmuhsu-18obj-test-zone2999falseobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.515 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:42.515 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:17:42.516 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?acl 2023-09-29T22:17:42.516 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?acl 2023-09-29T22:17:42.516 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.516 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?acl 2023-09-29T22:17:42.516 DEBUG:boto:Data: 2023-09-29T22:17:42.516 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.516 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:17:42.516 DEBUG:boto:Port: 8000 2023-09-29T22:17:42.516 DEBUG:boto:Params: {} 2023-09-29T22:17:42.516 DEBUG:boto:Token: None 2023-09-29T22:17:42.517 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-18/obj-test-zone2?acl 2023-09-29T22:17:42.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:98ngTZUkz3WT7os0CRV+/XlIfRg= 2023-09-29T22:17:42.517 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:98ngTZUkz3WT7os0CRV+/XlIfRg=', 'Content-Length': '0'} 2023-09-29T22:17:42.524 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a40032d98233af0-0065174d06-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.524 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.524 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.524 DEBUG:boto:Method: HEAD 2023-09-29T22:17:42.524 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:17:42.524 DEBUG:boto:Data: 2023-09-29T22:17:42.524 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.524 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.525 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.525 DEBUG:boto:Params: {} 2023-09-29T22:17:42.525 DEBUG:boto:Token: None 2023-09-29T22:17:42.525 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/ 2023-09-29T22:17:42.525 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU= 2023-09-29T22:17:42.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OXT/9HaZZ8MvjDrVWJTRMWQ0veU=', 'Content-Length': '0'} 2023-09-29T22:17:42.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000047fd229357e011d-0065174d06-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.527 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:17:42.528 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:17:42.528 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.528 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.528 DEBUG:boto:Method: GET 2023-09-29T22:17:42.528 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999&prefix=obj-test-zone3 2023-09-29T22:17:42.528 DEBUG:boto:Data: 2023-09-29T22:17:42.528 DEBUG:boto:Headers: {} 2023-09-29T22:17:42.528 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.528 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.529 DEBUG:boto:Params: {} 2023-09-29T22:17:42.529 DEBUG:boto:Token: None 2023-09-29T22:17:42.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:42 GMT /dmuhsu-18/?versions 2023-09-29T22:17:42.529 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY= 2023-09-29T22:17:42.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HUKmQq2I1guo75MsEjoDOQiSNAY=', 'Content-Length': '0'} 2023-09-29T22:17:42.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5adb6e108f134e-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.574 DEBUG:boto:b'dmuhsu-18obj-test-zone3999falseobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:42.575 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.575 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:17:42.575 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?acl 2023-09-29T22:17:42.575 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?acl 2023-09-29T22:17:42.575 DEBUG:boto:Method: PUT 2023-09-29T22:17:42.575 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?acl 2023-09-29T22:17:42.575 DEBUG:boto:Data: 2023-09-29T22:17:42.576 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:17:42.576 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:17:42.576 DEBUG:boto:Port: 8001 2023-09-29T22:17:42.576 DEBUG:boto:Params: {} 2023-09-29T22:17:42.576 DEBUG:boto:Token: None 2023-09-29T22:17:42.576 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:17:42 GMT x-amz-acl:public-read /dmuhsu-18/obj-test-zone3?acl 2023-09-29T22:17:42.576 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r813vSRUvIWBpSc1zjK8qj16RZQ= 2023-09-29T22:17:42.577 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r813vSRUvIWBpSc1zjK8qj16RZQ=', 'Content-Length': '0'} 2023-09-29T22:17:42.582 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001af7e13523e5d747-0065174d06-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:42.582 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-16 2023-09-29T22:17:42.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:42.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:42.582 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:42.642 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:42.642 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:42.678 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:42.678+0000 7fa500721880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:42.679 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.6, , 00000000003.14.10, , 00000000007.21.11, , , , , ] 2023-09-29T22:17:42.679 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:47.680 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:47.680+0000 7fa500721880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:47.680 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.6, , 00000000003.14.10, , 00000000007.21.11, , , , , ] 2023-09-29T22:17:47.680 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:52.681 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:52.681+0000 7fa500721880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:52.682 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.6, , 00000000003.14.10, , 00000000007.21.11, , , , , ] 2023-09-29T22:17:52.682 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:57.683 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:57.682+0000 7fa500721880 1 bucket sync caught up with source: 2023-09-29T22:17:57.683 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:57.683 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:57.683 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:17:57.682+0000 7fa500721880 0 bucket checkpoint complete 2023-09-29T22:17:57.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-16 2023-09-29T22:17:57.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:57.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:57.699 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:57.759 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:57.759 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:57.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:17:57.796+0000 7f7968413880 1 waiting for incremental sync to catch up: 2023-09-29T22:17:57.797 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.26.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:17:57.797 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:18:02.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:02.798+0000 7f7968413880 1 bucket sync caught up with source: 2023-09-29T22:18:02.798 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:18:02.799 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.29.6, , 00000000003.14.10, , 00000000008.23.10, , , , , ] 2023-09-29T22:18:02.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:02.798+0000 7f7968413880 0 bucket checkpoint complete 2023-09-29T22:18:02.813 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-16 2023-09-29T22:18:02.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:02.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:02.814 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:02.871 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:02.871 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:02.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:02.909+0000 7f92247a0880 1 bucket sync caught up with source: 2023-09-29T22:18:02.909 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000011.58.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:18:02.909 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000011.58.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:18:02.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:02.909+0000 7f92247a0880 0 bucket checkpoint complete 2023-09-29T22:18:02.921 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-16 2023-09-29T22:18:02.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:02.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:02.922 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:02.974 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:02.974 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:03.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:03.007+0000 7f906cc10880 1 bucket sync caught up with source: 2023-09-29T22:18:03.007 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.26.10, , 00000000011.58.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:18:03.007 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.26.10, , 00000000011.58.6, , 00000000007.49.10, , , , , ] 2023-09-29T22:18:03.008 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:03.007+0000 7f906cc10880 0 bucket checkpoint complete 2023-09-29T22:18:03.019 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-16 2023-09-29T22:18:03.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.019 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:03.070 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:03.071 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:03.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:03.103+0000 7f2300a15880 1 bucket sync caught up with source: 2023-09-29T22:18:03.103 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:03.103 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:03.103 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:18:03.103+0000 7f2300a15880 0 bucket checkpoint complete 2023-09-29T22:18:03.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-16 2023-09-29T22:18:03.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.116 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:03.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:03.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:03.216 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:03.215+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:03.216 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:03.216 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:08.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:08.217+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:08.217 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:08.217 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:13.218 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:13.218+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:13.218 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:13.219 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:18.219 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:18.219+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:18.220 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:18.220 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:23.221 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:23.220+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:23.221 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:23.221 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:28.222 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:28.222+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:28.222 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:28.223 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:33.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:33.223+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:33.224 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:33.224 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:38.225 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:38.225+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:38.225 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:38.225 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:43.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:43.226+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:43.227 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:43.227 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:48.228 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:48.227+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:48.228 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:48.228 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:53.229 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:53.229+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:53.229 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:53.229 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:18:58.230 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:18:58.230+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:18:58.231 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:18:58.231 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:03.232 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:03.231+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:19:03.232 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:19:03.232 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:08.233 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:08.233+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:19:08.233 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:19:08.233 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:13.234 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:13.234+0000 7fc980244880 1 waiting for incremental sync to catch up: 2023-09-29T22:19:13.235 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000013.62.6, , , , , ] 2023-09-29T22:19:13.235 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:18.238 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:18.235+0000 7fc980244880 1 bucket sync caught up with source: 2023-09-29T22:19:18.238 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:18.238 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.40.10, , 00000000014.65.6, , , , , ] 2023-09-29T22:19:18.239 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:18.235+0000 7fc980244880 0 bucket checkpoint complete 2023-09-29T22:19:18.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone2, test-zone1} 2023-09-29T22:19:18.252 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.252 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.252 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.252 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:18.252 DEBUG:boto:Data: 2023-09-29T22:19:18.252 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.252 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.253 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.253 DEBUG:boto:Params: {} 2023-09-29T22:19:18.253 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:19:18.253 DEBUG:boto:Token: None 2023-09-29T22:19:18.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.254 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T+kkjLGsDozxWML9AXqHSEdkPuI= 2023-09-29T22:19:18.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T+kkjLGsDozxWML9AXqHSEdkPuI=', 'Content-Length': '0'} 2023-09-29T22:19:18.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bceaf3add69f4944-0065174d66-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.259 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.259 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.259 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.259 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:18.259 DEBUG:boto:Data: 2023-09-29T22:19:18.259 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.259 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.259 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.259 DEBUG:boto:Params: {} 2023-09-29T22:19:18.260 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:19:18.260 DEBUG:boto:Token: None 2023-09-29T22:19:18.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T+kkjLGsDozxWML9AXqHSEdkPuI= 2023-09-29T22:19:18.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T+kkjLGsDozxWML9AXqHSEdkPuI=', 'Content-Length': '0'} 2023-09-29T22:19:18.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df71e664dfd5bde1-0065174d66-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:18.265 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.265 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.265 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.265 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.265 DEBUG:boto:Method: GET 2023-09-29T22:19:18.265 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.266 DEBUG:boto:Data: 2023-09-29T22:19:18.266 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.266 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.266 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.266 DEBUG:boto:Params: {} 2023-09-29T22:19:18.266 DEBUG:boto:Token: None 2023-09-29T22:19:18.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/?versions 2023-09-29T22:19:18.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q= 2023-09-29T22:19:18.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q=', 'Content-Length': '0'} 2023-09-29T22:19:18.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad65b01d28a6ce2-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.269 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.270 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:18.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:18.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:18.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:18.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:18.271 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.271 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.271 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.271 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.271 DEBUG:boto:Method: GET 2023-09-29T22:19:18.271 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.271 DEBUG:boto:Data: 2023-09-29T22:19:18.271 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.272 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.272 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.272 DEBUG:boto:Params: {} 2023-09-29T22:19:18.272 DEBUG:boto:Token: None 2023-09-29T22:19:18.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/?versions 2023-09-29T22:19:18.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q= 2023-09-29T22:19:18.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q=', 'Content-Length': '0'} 2023-09-29T22:19:18.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586f75169923bd4f-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.275 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.276 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:18.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:18.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:18.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:18.276 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.276 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.277 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.277 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.277 DEBUG:boto:Method: GET 2023-09-29T22:19:18.277 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.277 DEBUG:boto:Data: 2023-09-29T22:19:18.277 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.277 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.277 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.277 DEBUG:boto:Params: {} 2023-09-29T22:19:18.278 DEBUG:boto:Token: None 2023-09-29T22:19:18.278 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/?versions 2023-09-29T22:19:18.278 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q= 2023-09-29T22:19:18.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q=', 'Content-Length': '0'} 2023-09-29T22:19:18.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37a7ab90e37f05b-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.281 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.281 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.281 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.281 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.282 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.282 DEBUG:boto:Method: GET 2023-09-29T22:19:18.282 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:18.282 DEBUG:boto:Data: 2023-09-29T22:19:18.282 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.282 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.282 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.282 DEBUG:boto:Params: {} 2023-09-29T22:19:18.283 DEBUG:boto:Token: None 2023-09-29T22:19:18.283 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/?versions 2023-09-29T22:19:18.283 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q= 2023-09-29T22:19:18.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GauttyHTXy/U4ZpP5hNACMo+3+Q=', 'Content-Length': '0'} 2023-09-29T22:19:18.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf28c05f1c5531d-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.286 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:18.287 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.287 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.287 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.288 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.288 DEBUG:boto:Method: GET 2023-09-29T22:19:18.288 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.288 DEBUG:boto:Data: 2023-09-29T22:19:18.288 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.288 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.288 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.288 DEBUG:boto:Params: {} 2023-09-29T22:19:18.288 DEBUG:boto:Token: None 2023-09-29T22:19:18.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.289 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000907d22494cb9e57f-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.291 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.291 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.291 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.291 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.291 DEBUG:boto:Method: GET 2023-09-29T22:19:18.291 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.292 DEBUG:boto:Data: 2023-09-29T22:19:18.292 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.292 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.292 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.292 DEBUG:boto:Params: {} 2023-09-29T22:19:18.292 DEBUG:boto:Token: None 2023-09-29T22:19:18.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b495e128658d95a1-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.295 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.295 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.295 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.295 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.295 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.295 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.295 DEBUG:boto:Data: 2023-09-29T22:19:18.295 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.295 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.295 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.296 DEBUG:boto:Params: {} 2023-09-29T22:19:18.296 DEBUG:boto:Token: None 2023-09-29T22:19:18.296 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+4FzGBPvILDR7BpXz5mO6pd0BU= 2023-09-29T22:19:18.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+4FzGBPvILDR7BpXz5mO6pd0BU=', 'Content-Length': '0'} 2023-09-29T22:19:18.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000728ce3c4b549220a-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.298 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.298 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.298 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.299 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.299 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.299 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.299 DEBUG:boto:Data: 2023-09-29T22:19:18.299 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.299 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.299 DEBUG:boto:Params: {} 2023-09-29T22:19:18.299 DEBUG:boto:Token: None 2023-09-29T22:19:18.300 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.300 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+4FzGBPvILDR7BpXz5mO6pd0BU= 2023-09-29T22:19:18.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+4FzGBPvILDR7BpXz5mO6pd0BU=', 'Content-Length': '0'} 2023-09-29T22:19:18.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016f954e93a866136-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:18.302 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.302 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.302 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.302 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.302 DEBUG:boto:Method: GET 2023-09-29T22:19:18.302 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.303 DEBUG:boto:Data: 2023-09-29T22:19:18.303 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.303 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.303 DEBUG:boto:Params: {} 2023-09-29T22:19:18.303 DEBUG:boto:Token: None 2023-09-29T22:19:18.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007200e0b7674098b9-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.305 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.306 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.306 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.306 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.306 DEBUG:boto:Method: GET 2023-09-29T22:19:18.306 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.306 DEBUG:boto:Data: 2023-09-29T22:19:18.306 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.306 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.306 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.307 DEBUG:boto:Params: {} 2023-09-29T22:19:18.307 DEBUG:boto:Token: None 2023-09-29T22:19:18.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb62346b6b38f6f8-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.309 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.309 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.309 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.309 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.310 DEBUG:boto:Method: GET 2023-09-29T22:19:18.310 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.310 DEBUG:boto:Data: 2023-09-29T22:19:18.310 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.310 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.310 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.310 DEBUG:boto:Params: {} 2023-09-29T22:19:18.310 DEBUG:boto:Token: None 2023-09-29T22:19:18.310 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.311 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000268193b17009309a-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.313 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.313 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.313 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.314 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.314 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.314 DEBUG:boto:Method: GET 2023-09-29T22:19:18.314 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.314 DEBUG:boto:Data: 2023-09-29T22:19:18.314 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.314 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.314 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.314 DEBUG:boto:Params: {} 2023-09-29T22:19:18.315 DEBUG:boto:Token: None 2023-09-29T22:19:18.315 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.315 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ab203432f59b69-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.358 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.359 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.359 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.359 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.359 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:18.359 DEBUG:boto:Data: 2023-09-29T22:19:18.360 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.360 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.360 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.360 DEBUG:boto:Params: {} 2023-09-29T22:19:18.360 DEBUG:boto:Token: None 2023-09-29T22:19:18.360 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy 2023-09-29T22:19:18.361 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q5OBIIUQXYVutzLAEI/4jSZG31A= 2023-09-29T22:19:18.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q5OBIIUQXYVutzLAEI/4jSZG31A=', 'Content-Length': '0'} 2023-09-29T22:19:18.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ac995f4c603d013-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.364 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.364 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.364 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.364 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:18.364 DEBUG:boto:Data: 2023-09-29T22:19:18.365 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.365 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.365 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.365 DEBUG:boto:Params: {} 2023-09-29T22:19:18.365 DEBUG:boto:Token: None 2023-09-29T22:19:18.365 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy 2023-09-29T22:19:18.365 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q5OBIIUQXYVutzLAEI/4jSZG31A= 2023-09-29T22:19:18.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q5OBIIUQXYVutzLAEI/4jSZG31A=', 'Content-Length': '0'} 2023-09-29T22:19:18.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f06bc83c26a25e16-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:18.368 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.368 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.368 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.368 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.368 DEBUG:boto:Method: GET 2023-09-29T22:19:18.368 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.368 DEBUG:boto:Data: 2023-09-29T22:19:18.368 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.369 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.369 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.369 DEBUG:boto:Params: {} 2023-09-29T22:19:18.369 DEBUG:boto:Token: None 2023-09-29T22:19:18.369 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.369 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000268b4b13ebaf9a9d-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.371 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:18.371 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:18.371 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.372 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.372 DEBUG:boto:Method: GET 2023-09-29T22:19:18.372 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.372 DEBUG:boto:Data: 2023-09-29T22:19:18.372 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.372 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.372 DEBUG:boto:Params: {} 2023-09-29T22:19:18.373 DEBUG:boto:Token: None 2023-09-29T22:19:18.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:18.373 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU= 2023-09-29T22:19:18.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qHsrzNqkxdeREk7+BLqg4T/vbuU=', 'Content-Length': '0'} 2023-09-29T22:19:18.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001597db73f3322354-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.375 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.375 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.375 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.375 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.375 DEBUG:boto:Method: GET 2023-09-29T22:19:18.376 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.376 DEBUG:boto:Data: 2023-09-29T22:19:18.376 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.376 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.376 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.376 DEBUG:boto:Params: {} 2023-09-29T22:19:18.376 DEBUG:boto:Token: None 2023-09-29T22:19:18.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.377 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191b030e945fa917-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.422 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.423 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.423 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.423 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.424 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.424 DEBUG:boto:Method: GET 2023-09-29T22:19:18.424 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:18.424 DEBUG:boto:Data: 2023-09-29T22:19:18.424 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.424 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.424 DEBUG:boto:Params: {} 2023-09-29T22:19:18.425 DEBUG:boto:Token: None 2023-09-29T22:19:18.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b9e3f3ab61b3f19-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.428 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:18.428 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.429 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.429 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.429 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.429 DEBUG:boto:Method: GET 2023-09-29T22:19:18.429 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.429 DEBUG:boto:Data: 2023-09-29T22:19:18.429 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.429 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.429 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.429 DEBUG:boto:Params: {} 2023-09-29T22:19:18.430 DEBUG:boto:Token: None 2023-09-29T22:19:18.430 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.430 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023eb5b39d1b77342-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.478 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.479 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.479 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.479 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.479 DEBUG:boto:Method: GET 2023-09-29T22:19:18.479 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.479 DEBUG:boto:Data: 2023-09-29T22:19:18.479 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.480 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.480 DEBUG:boto:Params: {} 2023-09-29T22:19:18.480 DEBUG:boto:Token: None 2023-09-29T22:19:18.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9dd0c4fd736518e-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.483 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.483 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.483 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.483 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.483 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.483 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.484 DEBUG:boto:Data: 2023-09-29T22:19:18.484 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.484 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.484 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.484 DEBUG:boto:Params: {} 2023-09-29T22:19:18.484 DEBUG:boto:Token: None 2023-09-29T22:19:18.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QGjbPU27IqIBlJ7bwEBflXcK7Hk= 2023-09-29T22:19:18.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QGjbPU27IqIBlJ7bwEBflXcK7Hk=', 'Content-Length': '0'} 2023-09-29T22:19:18.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed95176c781910b2-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.486 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.487 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.487 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.487 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.487 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.487 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.487 DEBUG:boto:Data: 2023-09-29T22:19:18.487 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.488 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.488 DEBUG:boto:Params: {} 2023-09-29T22:19:18.488 DEBUG:boto:Token: None 2023-09-29T22:19:18.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QGjbPU27IqIBlJ7bwEBflXcK7Hk= 2023-09-29T22:19:18.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QGjbPU27IqIBlJ7bwEBflXcK7Hk=', 'Content-Length': '0'} 2023-09-29T22:19:18.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002aa494ff8bd27f64-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:18.491 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.491 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.491 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.491 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.491 DEBUG:boto:Method: GET 2023-09-29T22:19:18.491 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.491 DEBUG:boto:Data: 2023-09-29T22:19:18.491 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.492 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.492 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.492 DEBUG:boto:Params: {} 2023-09-29T22:19:18.492 DEBUG:boto:Token: None 2023-09-29T22:19:18.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000389db6a0120b679e-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.538 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.539 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.539 DEBUG:boto:Method: GET 2023-09-29T22:19:18.539 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.539 DEBUG:boto:Data: 2023-09-29T22:19:18.539 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.540 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.540 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.540 DEBUG:boto:Params: {} 2023-09-29T22:19:18.540 DEBUG:boto:Token: None 2023-09-29T22:19:18.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b526329268ee5cd9-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.543 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.543 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.543 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.543 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.543 DEBUG:boto:Method: GET 2023-09-29T22:19:18.543 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.544 DEBUG:boto:Data: 2023-09-29T22:19:18.544 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.544 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.544 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.544 DEBUG:boto:Params: {} 2023-09-29T22:19:18.544 DEBUG:boto:Token: None 2023-09-29T22:19:18.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1edfd4ed76dbb99-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.547 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.547 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.548 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.548 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.548 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.548 DEBUG:boto:Method: GET 2023-09-29T22:19:18.548 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.548 DEBUG:boto:Data: 2023-09-29T22:19:18.548 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.548 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.549 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.549 DEBUG:boto:Params: {} 2023-09-29T22:19:18.549 DEBUG:boto:Token: None 2023-09-29T22:19:18.549 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e999e9fb4947dc2-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.552 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.552 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.552 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.552 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.553 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.553 DEBUG:boto:Data: 2023-09-29T22:19:18.553 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.553 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.553 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.553 DEBUG:boto:Params: {} 2023-09-29T22:19:18.553 DEBUG:boto:Token: None 2023-09-29T22:19:18.553 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.554 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vHmF2Sv5kG50JkudgcmpysCM7rI= 2023-09-29T22:19:18.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vHmF2Sv5kG50JkudgcmpysCM7rI=', 'Content-Length': '0'} 2023-09-29T22:19:18.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5fba6a24105c845-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.556 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.556 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.557 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.557 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.557 DEBUG:boto:Data: 2023-09-29T22:19:18.557 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.557 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.557 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.557 DEBUG:boto:Params: {} 2023-09-29T22:19:18.557 DEBUG:boto:Token: None 2023-09-29T22:19:18.557 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.558 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vHmF2Sv5kG50JkudgcmpysCM7rI= 2023-09-29T22:19:18.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vHmF2Sv5kG50JkudgcmpysCM7rI=', 'Content-Length': '0'} 2023-09-29T22:19:18.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000772281ae98578a62-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:18.560 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.561 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.561 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.561 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.561 DEBUG:boto:Method: GET 2023-09-29T22:19:18.561 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.561 DEBUG:boto:Data: 2023-09-29T22:19:18.561 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.561 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.562 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.562 DEBUG:boto:Params: {} 2023-09-29T22:19:18.562 DEBUG:boto:Token: None 2023-09-29T22:19:18.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.562 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d4e5869cfe48ed2-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.610 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.611 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:18.611 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.611 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.611 DEBUG:boto:Method: GET 2023-09-29T22:19:18.611 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.611 DEBUG:boto:Data: 2023-09-29T22:19:18.611 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.611 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.611 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.612 DEBUG:boto:Params: {} 2023-09-29T22:19:18.612 DEBUG:boto:Token: None 2023-09-29T22:19:18.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:18.612 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE= 2023-09-29T22:19:18.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5NSBfRfjfSa+U5TMabXelcjEoeE=', 'Content-Length': '0'} 2023-09-29T22:19:18.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d27eb92db167380d-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.614 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.615 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.615 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.615 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.615 DEBUG:boto:Method: GET 2023-09-29T22:19:18.615 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.615 DEBUG:boto:Data: 2023-09-29T22:19:18.615 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.615 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.615 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.615 DEBUG:boto:Params: {} 2023-09-29T22:19:18.616 DEBUG:boto:Token: None 2023-09-29T22:19:18.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004115cd647868d442-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.662 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.663 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.663 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.664 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.664 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.664 DEBUG:boto:Method: GET 2023-09-29T22:19:18.664 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:18.664 DEBUG:boto:Data: 2023-09-29T22:19:18.664 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.664 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.665 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.665 DEBUG:boto:Params: {} 2023-09-29T22:19:18.665 DEBUG:boto:Token: None 2023-09-29T22:19:18.665 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.665 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015738a9c53547e64-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.668 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:18.671 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.671 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.671 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.671 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.671 DEBUG:boto:Method: GET 2023-09-29T22:19:18.672 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.672 DEBUG:boto:Data: 2023-09-29T22:19:18.672 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.672 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.672 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.672 DEBUG:boto:Params: {} 2023-09-29T22:19:18.673 DEBUG:boto:Token: None 2023-09-29T22:19:18.673 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.673 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b0b4bb3b91f2b0-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.676 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.676 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.676 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.676 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.676 DEBUG:boto:Method: GET 2023-09-29T22:19:18.676 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.677 DEBUG:boto:Data: 2023-09-29T22:19:18.677 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.677 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.677 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.677 DEBUG:boto:Params: {} 2023-09-29T22:19:18.677 DEBUG:boto:Token: None 2023-09-29T22:19:18.678 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.678 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007050bf67c94b4783-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.726 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.727 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.727 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.727 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.727 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.727 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.727 DEBUG:boto:Data: 2023-09-29T22:19:18.727 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.727 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.728 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.728 DEBUG:boto:Params: {} 2023-09-29T22:19:18.728 DEBUG:boto:Token: None 2023-09-29T22:19:18.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bRSBQCXhHBmvdlug4jKKnNhQAYo= 2023-09-29T22:19:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bRSBQCXhHBmvdlug4jKKnNhQAYo=', 'Content-Length': '0'} 2023-09-29T22:19:18.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c74f155294c8be46-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.731 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.731 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.731 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.732 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.732 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.732 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.732 DEBUG:boto:Data: 2023-09-29T22:19:18.732 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.732 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.733 DEBUG:boto:Params: {} 2023-09-29T22:19:18.733 DEBUG:boto:Token: None 2023-09-29T22:19:18.733 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.733 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bRSBQCXhHBmvdlug4jKKnNhQAYo= 2023-09-29T22:19:18.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bRSBQCXhHBmvdlug4jKKnNhQAYo=', 'Content-Length': '0'} 2023-09-29T22:19:18.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e1440778eda7861-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:18.736 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.736 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.736 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.736 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.736 DEBUG:boto:Method: GET 2023-09-29T22:19:18.737 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.737 DEBUG:boto:Data: 2023-09-29T22:19:18.737 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.737 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.737 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.737 DEBUG:boto:Params: {} 2023-09-29T22:19:18.738 DEBUG:boto:Token: None 2023-09-29T22:19:18.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.738 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c72c3b21350e5753-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.740 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.740 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.740 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.740 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.740 DEBUG:boto:Method: GET 2023-09-29T22:19:18.741 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.741 DEBUG:boto:Data: 2023-09-29T22:19:18.741 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.741 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.741 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.741 DEBUG:boto:Params: {} 2023-09-29T22:19:18.741 DEBUG:boto:Token: None 2023-09-29T22:19:18.741 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.741 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099414a7bb5ff6496-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.786 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.787 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.787 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.787 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.787 DEBUG:boto:Method: GET 2023-09-29T22:19:18.787 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.787 DEBUG:boto:Data: 2023-09-29T22:19:18.787 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.787 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.788 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.788 DEBUG:boto:Params: {} 2023-09-29T22:19:18.788 DEBUG:boto:Token: None 2023-09-29T22:19:18.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4b46fdc717e6bd-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.791 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.792 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.792 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.792 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.792 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.792 DEBUG:boto:Method: GET 2023-09-29T22:19:18.792 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.792 DEBUG:boto:Data: 2023-09-29T22:19:18.793 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.793 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.793 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.793 DEBUG:boto:Params: {} 2023-09-29T22:19:18.793 DEBUG:boto:Token: None 2023-09-29T22:19:18.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.794 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000263a1b70c8d72a86-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.842 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.843 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.843 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.843 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.843 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.843 DEBUG:boto:Data: 2023-09-29T22:19:18.843 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.843 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.844 DEBUG:boto:Params: {} 2023-09-29T22:19:18.844 DEBUG:boto:Token: None 2023-09-29T22:19:18.845 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sHV6KyBGzC3AQQoI5n3h9eNjxRw= 2023-09-29T22:19:18.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sHV6KyBGzC3AQQoI5n3h9eNjxRw=', 'Content-Length': '0'} 2023-09-29T22:19:18.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4c5ee0a0a09ba19-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.850 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.850 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.850 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.850 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.851 DEBUG:boto:Data: 2023-09-29T22:19:18.851 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.851 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.851 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.851 DEBUG:boto:Params: {} 2023-09-29T22:19:18.851 DEBUG:boto:Token: None 2023-09-29T22:19:18.851 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.851 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sHV6KyBGzC3AQQoI5n3h9eNjxRw= 2023-09-29T22:19:18.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sHV6KyBGzC3AQQoI5n3h9eNjxRw=', 'Content-Length': '0'} 2023-09-29T22:19:18.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3337612e24f439e-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:18.854 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.854 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.854 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.854 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.854 DEBUG:boto:Method: GET 2023-09-29T22:19:18.855 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.855 DEBUG:boto:Data: 2023-09-29T22:19:18.855 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.855 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.855 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.855 DEBUG:boto:Params: {} 2023-09-29T22:19:18.855 DEBUG:boto:Token: None 2023-09-29T22:19:18.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.855 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d6903499d09df26-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.898 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.899 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:18.899 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.899 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.899 DEBUG:boto:Method: GET 2023-09-29T22:19:18.899 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.900 DEBUG:boto:Data: 2023-09-29T22:19:18.900 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.900 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.900 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.900 DEBUG:boto:Params: {} 2023-09-29T22:19:18.901 DEBUG:boto:Token: None 2023-09-29T22:19:18.901 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:18.901 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc= 2023-09-29T22:19:18.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LB49W5v6rC7uZWpfTw1SWWDI2yc=', 'Content-Length': '0'} 2023-09-29T22:19:18.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000db363f9c203e252-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.946 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.947 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.947 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.947 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.947 DEBUG:boto:Method: GET 2023-09-29T22:19:18.947 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.947 DEBUG:boto:Data: 2023-09-29T22:19:18.947 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.948 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.948 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.948 DEBUG:boto:Params: {} 2023-09-29T22:19:18.948 DEBUG:boto:Token: None 2023-09-29T22:19:18.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023a052bed6b00e67-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.952 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.952 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:18.953 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:18.953 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.953 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.953 DEBUG:boto:Method: GET 2023-09-29T22:19:18.953 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:18.953 DEBUG:boto:Data: 2023-09-29T22:19:18.953 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.954 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.954 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.954 DEBUG:boto:Params: {} 2023-09-29T22:19:18.954 DEBUG:boto:Token: None 2023-09-29T22:19:18.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/ 2023-09-29T22:19:18.955 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8= 2023-09-29T22:19:18.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZgbxW5/0b5nd7ev1ww+58qAgtr8=', 'Content-Length': '0'} 2023-09-29T22:19:18.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193f3166228c16ad-0065174d66-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.958 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:18.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:18.958 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.958 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.958 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.959 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.959 DEBUG:boto:Method: GET 2023-09-29T22:19:18.959 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.959 DEBUG:boto:Data: 2023-09-29T22:19:18.959 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.959 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.959 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.959 DEBUG:boto:Params: {} 2023-09-29T22:19:18.959 DEBUG:boto:Token: None 2023-09-29T22:19:18.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ= 2023-09-29T22:19:18.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ=', 'Content-Length': '0'} 2023-09-29T22:19:18.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab2aa05f4a65b538-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.962 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.963 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.963 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.963 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.963 DEBUG:boto:Method: GET 2023-09-29T22:19:18.963 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.963 DEBUG:boto:Data: 2023-09-29T22:19:18.963 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.963 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.964 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.964 DEBUG:boto:Params: {} 2023-09-29T22:19:18.964 DEBUG:boto:Token: None 2023-09-29T22:19:18.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ= 2023-09-29T22:19:18.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ=', 'Content-Length': '0'} 2023-09-29T22:19:18.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d64fd63f4db9d92c-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.966 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.967 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.967 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.967 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.967 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.967 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.967 DEBUG:boto:Data: 2023-09-29T22:19:18.967 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.968 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.968 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.968 DEBUG:boto:Params: {} 2023-09-29T22:19:18.968 DEBUG:boto:Token: None 2023-09-29T22:19:18.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QqUxurkChyWBOf1b4l6F1zElwG4= 2023-09-29T22:19:18.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QqUxurkChyWBOf1b4l6F1zElwG4=', 'Content-Length': '0'} 2023-09-29T22:19:18.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc9c55dbc523d85f-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.971 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.971 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.971 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.971 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.971 DEBUG:boto:Method: HEAD 2023-09-29T22:19:18.972 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.972 DEBUG:boto:Data: 2023-09-29T22:19:18.972 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:18.972 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.972 DEBUG:boto:Params: {} 2023-09-29T22:19:18.972 DEBUG:boto:Token: None 2023-09-29T22:19:18.972 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QqUxurkChyWBOf1b4l6F1zElwG4= 2023-09-29T22:19:18.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QqUxurkChyWBOf1b4l6F1zElwG4=', 'Content-Length': '0'} 2023-09-29T22:19:18.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c20fa25618ad406-0065174d66-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:18.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:18.975 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.975 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:18.975 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.975 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.975 DEBUG:boto:Method: GET 2023-09-29T22:19:18.976 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.976 DEBUG:boto:Data: 2023-09-29T22:19:18.976 DEBUG:boto:Headers: {} 2023-09-29T22:19:18.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:18.976 DEBUG:boto:Port: 8000 2023-09-29T22:19:18.976 DEBUG:boto:Params: {} 2023-09-29T22:19:18.977 DEBUG:boto:Token: None 2023-09-29T22:19:18.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:18 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:18.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ= 2023-09-29T22:19:18.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tHxinsvKNJCPEfXDHK4WsnwMIoQ=', 'Content-Length': '0'} 2023-09-29T22:19:18.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027d8d146f7c47fb1-0065174d66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.022 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.023 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.023 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.023 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.023 DEBUG:boto:Method: GET 2023-09-29T22:19:19.023 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.024 DEBUG:boto:Data: 2023-09-29T22:19:19.024 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.024 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.024 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.024 DEBUG:boto:Params: {} 2023-09-29T22:19:19.025 DEBUG:boto:Token: None 2023-09-29T22:19:19.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086bc333dc0a19e05-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.070 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.071 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.071 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.071 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.071 DEBUG:boto:Method: GET 2023-09-29T22:19:19.072 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.072 DEBUG:boto:Data: 2023-09-29T22:19:19.072 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.072 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:19.073 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.073 DEBUG:boto:Params: {} 2023-09-29T22:19:19.073 DEBUG:boto:Token: None 2023-09-29T22:19:19.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.074 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb23df6bf3d6bb0-0065174d67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.118 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.119 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.119 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.120 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.120 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.120 DEBUG:boto:Method: GET 2023-09-29T22:19:19.120 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.120 DEBUG:boto:Data: 2023-09-29T22:19:19.120 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.121 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.121 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.121 DEBUG:boto:Params: {} 2023-09-29T22:19:19.121 DEBUG:boto:Token: None 2023-09-29T22:19:19.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f507542b94496bb-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.124 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.125 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.125 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.125 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.125 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.125 DEBUG:boto:Data: 2023-09-29T22:19:19.125 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.125 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:19.125 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.126 DEBUG:boto:Params: {} 2023-09-29T22:19:19.126 DEBUG:boto:Token: None 2023-09-29T22:19:19.126 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.126 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs= 2023-09-29T22:19:19.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs=', 'Content-Length': '0'} 2023-09-29T22:19:19.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c35522018705f1c8-0065174d67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.129 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.129 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.129 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.129 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.129 DEBUG:boto:Data: 2023-09-29T22:19:19.130 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.130 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.130 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.130 DEBUG:boto:Params: {} 2023-09-29T22:19:19.130 DEBUG:boto:Token: None 2023-09-29T22:19:19.130 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.130 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs= 2023-09-29T22:19:19.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs=', 'Content-Length': '0'} 2023-09-29T22:19:19.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d45ffa9a80ff2c2-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:19.133 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.133 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.133 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.134 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.134 DEBUG:boto:Method: GET 2023-09-29T22:19:19.134 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.134 DEBUG:boto:Data: 2023-09-29T22:19:19.134 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.134 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:19.134 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.134 DEBUG:boto:Params: {} 2023-09-29T22:19:19.135 DEBUG:boto:Token: None 2023-09-29T22:19:19.135 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c36d18d3516672e-0065174d67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.182 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.183 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.183 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.183 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.183 DEBUG:boto:Method: GET 2023-09-29T22:19:19.183 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.184 DEBUG:boto:Data: 2023-09-29T22:19:19.184 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.184 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.184 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.184 DEBUG:boto:Params: {} 2023-09-29T22:19:19.184 DEBUG:boto:Token: None 2023-09-29T22:19:19.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.185 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fea654fdb99662c3-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.234 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.235 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.235 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.235 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.235 DEBUG:boto:Method: GET 2023-09-29T22:19:19.235 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.235 DEBUG:boto:Data: 2023-09-29T22:19:19.235 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.235 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:19.235 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.235 DEBUG:boto:Params: {} 2023-09-29T22:19:19.236 DEBUG:boto:Token: None 2023-09-29T22:19:19.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.236 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d2f87a98233f77-0065174d67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.239 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.239 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.239 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.239 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.240 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.240 DEBUG:boto:Method: GET 2023-09-29T22:19:19.240 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.240 DEBUG:boto:Data: 2023-09-29T22:19:19.240 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.240 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.240 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.240 DEBUG:boto:Params: {} 2023-09-29T22:19:19.240 DEBUG:boto:Token: None 2023-09-29T22:19:19.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.241 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aec5f1cbd01bfd6-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.286 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.287 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone2, test-zone1} 2023-09-29T22:19:19.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone3, test-zone1} 2023-09-29T22:19:19.287 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.287 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.287 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.287 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:19.287 DEBUG:boto:Data: 2023-09-29T22:19:19.287 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.288 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.288 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.288 DEBUG:boto:Params: {} 2023-09-29T22:19:19.288 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:19:19.288 DEBUG:boto:Token: None 2023-09-29T22:19:19.288 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZCY5leU563qlqxtVrye5A6VVc+I= 2023-09-29T22:19:19.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZCY5leU563qlqxtVrye5A6VVc+I=', 'Content-Length': '0'} 2023-09-29T22:19:19.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000346e9a1dc0d69fc0-0065174d67-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.292 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.292 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.292 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.292 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:19.293 DEBUG:boto:Data: 2023-09-29T22:19:19.293 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.293 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.293 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.293 DEBUG:boto:Params: {} 2023-09-29T22:19:19.293 DEBUG:boto:Token: None 2023-09-29T22:19:19.293 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.293 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZCY5leU563qlqxtVrye5A6VVc+I= 2023-09-29T22:19:19.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZCY5leU563qlqxtVrye5A6VVc+I=', 'Content-Length': '0'} 2023-09-29T22:19:19.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053b07722fdfab58e-0065174d67-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:19.296 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.296 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.296 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.296 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.296 DEBUG:boto:Method: GET 2023-09-29T22:19:19.296 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.296 DEBUG:boto:Data: 2023-09-29T22:19:19.296 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.296 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.296 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.297 DEBUG:boto:Params: {} 2023-09-29T22:19:19.297 DEBUG:boto:Token: None 2023-09-29T22:19:19.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/?versions 2023-09-29T22:19:19.297 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk= 2023-09-29T22:19:19.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk=', 'Content-Length': '0'} 2023-09-29T22:19:19.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000847e647379f27755-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.300 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:19.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:19.301 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:19.301 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.301 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.301 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.301 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.301 DEBUG:boto:Method: GET 2023-09-29T22:19:19.301 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.301 DEBUG:boto:Data: 2023-09-29T22:19:19.301 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.301 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.302 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.302 DEBUG:boto:Params: {} 2023-09-29T22:19:19.302 DEBUG:boto:Token: None 2023-09-29T22:19:19.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/?versions 2023-09-29T22:19:19.302 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk= 2023-09-29T22:19:19.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk=', 'Content-Length': '0'} 2023-09-29T22:19:19.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccab091f2f1ee262-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.305 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.305 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:19.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:19.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:19.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:19.306 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.306 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.306 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.306 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.306 DEBUG:boto:Method: GET 2023-09-29T22:19:19.306 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.306 DEBUG:boto:Data: 2023-09-29T22:19:19.306 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.306 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.306 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.307 DEBUG:boto:Params: {} 2023-09-29T22:19:19.307 DEBUG:boto:Token: None 2023-09-29T22:19:19.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/?versions 2023-09-29T22:19:19.307 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk= 2023-09-29T22:19:19.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk=', 'Content-Length': '0'} 2023-09-29T22:19:19.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdbc878071d9502a-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.310 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.310 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.310 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.310 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.310 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.310 DEBUG:boto:Method: GET 2023-09-29T22:19:19.311 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:19.311 DEBUG:boto:Data: 2023-09-29T22:19:19.311 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.311 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.311 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.311 DEBUG:boto:Params: {} 2023-09-29T22:19:19.311 DEBUG:boto:Token: None 2023-09-29T22:19:19.311 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/?versions 2023-09-29T22:19:19.311 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk= 2023-09-29T22:19:19.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFJYwNYE7D1lpex0w0qDJUvrRkk=', 'Content-Length': '0'} 2023-09-29T22:19:19.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ac6f37c8c23a95-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.314 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:19.315 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.315 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.315 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.315 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.315 DEBUG:boto:Method: GET 2023-09-29T22:19:19.315 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.316 DEBUG:boto:Data: 2023-09-29T22:19:19.316 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.316 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.316 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.316 DEBUG:boto:Params: {} 2023-09-29T22:19:19.316 DEBUG:boto:Token: None 2023-09-29T22:19:19.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e7867318b9cdbc5-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.318 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.318 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.319 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.319 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.319 DEBUG:boto:Method: GET 2023-09-29T22:19:19.319 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.319 DEBUG:boto:Data: 2023-09-29T22:19:19.319 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.319 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.319 DEBUG:boto:Params: {} 2023-09-29T22:19:19.320 DEBUG:boto:Token: None 2023-09-29T22:19:19.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4b573df16c2689a-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.322 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.322 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.322 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.322 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.322 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.322 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.323 DEBUG:boto:Data: 2023-09-29T22:19:19.323 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.323 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.323 DEBUG:boto:Params: {} 2023-09-29T22:19:19.323 DEBUG:boto:Token: None 2023-09-29T22:19:19.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jeO+15G1GMP5bMUztJvx3q+rvzI= 2023-09-29T22:19:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jeO+15G1GMP5bMUztJvx3q+rvzI=', 'Content-Length': '0'} 2023-09-29T22:19:19.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd14b6d7619083a4-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.325 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.326 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.326 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.326 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.326 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.326 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.326 DEBUG:boto:Data: 2023-09-29T22:19:19.326 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.326 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.326 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.327 DEBUG:boto:Params: {} 2023-09-29T22:19:19.327 DEBUG:boto:Token: None 2023-09-29T22:19:19.327 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jeO+15G1GMP5bMUztJvx3q+rvzI= 2023-09-29T22:19:19.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jeO+15G1GMP5bMUztJvx3q+rvzI=', 'Content-Length': '0'} 2023-09-29T22:19:19.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095135a1c7eb70631-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:19.329 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.329 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.329 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.329 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.330 DEBUG:boto:Method: GET 2023-09-29T22:19:19.330 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.330 DEBUG:boto:Data: 2023-09-29T22:19:19.330 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.330 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.330 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.330 DEBUG:boto:Params: {} 2023-09-29T22:19:19.330 DEBUG:boto:Token: None 2023-09-29T22:19:19.331 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e29c0451d752aea2-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.333 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.333 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.333 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.333 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.333 DEBUG:boto:Method: GET 2023-09-29T22:19:19.333 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.333 DEBUG:boto:Data: 2023-09-29T22:19:19.334 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.334 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.334 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.334 DEBUG:boto:Params: {} 2023-09-29T22:19:19.334 DEBUG:boto:Token: None 2023-09-29T22:19:19.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.334 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000818dc4a9fea02792-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.336 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.336 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.336 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.336 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.337 DEBUG:boto:Method: GET 2023-09-29T22:19:19.337 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.337 DEBUG:boto:Data: 2023-09-29T22:19:19.337 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.337 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.337 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.337 DEBUG:boto:Params: {} 2023-09-29T22:19:19.337 DEBUG:boto:Token: None 2023-09-29T22:19:19.337 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eaaf195770b0f79-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.340 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.340 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.340 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.341 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.341 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.341 DEBUG:boto:Method: GET 2023-09-29T22:19:19.341 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.341 DEBUG:boto:Data: 2023-09-29T22:19:19.341 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.341 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.341 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.341 DEBUG:boto:Params: {} 2023-09-29T22:19:19.342 DEBUG:boto:Token: None 2023-09-29T22:19:19.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6367ba3cd08159f-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.386 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.387 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.387 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.387 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.387 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:19.387 DEBUG:boto:Data: 2023-09-29T22:19:19.387 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.387 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.387 DEBUG:boto:Params: {} 2023-09-29T22:19:19.388 DEBUG:boto:Token: None 2023-09-29T22:19:19.388 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy 2023-09-29T22:19:19.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uZqRn23cMEqGNAn9sXvfrk5m38o= 2023-09-29T22:19:19.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uZqRn23cMEqGNAn9sXvfrk5m38o=', 'Content-Length': '0'} 2023-09-29T22:19:19.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb44a3b2c19fbe9d-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.391 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.391 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.391 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.391 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:19.391 DEBUG:boto:Data: 2023-09-29T22:19:19.391 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.391 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.391 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.391 DEBUG:boto:Params: {} 2023-09-29T22:19:19.392 DEBUG:boto:Token: None 2023-09-29T22:19:19.392 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy 2023-09-29T22:19:19.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uZqRn23cMEqGNAn9sXvfrk5m38o= 2023-09-29T22:19:19.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uZqRn23cMEqGNAn9sXvfrk5m38o=', 'Content-Length': '0'} 2023-09-29T22:19:19.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f6d971eaa04fb09-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:19.394 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.394 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.394 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.395 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.395 DEBUG:boto:Method: GET 2023-09-29T22:19:19.395 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.395 DEBUG:boto:Data: 2023-09-29T22:19:19.395 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.395 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.395 DEBUG:boto:Params: {} 2023-09-29T22:19:19.395 DEBUG:boto:Token: None 2023-09-29T22:19:19.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b1915632b3483e5-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.398 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:19.398 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:19.398 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.398 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.398 DEBUG:boto:Method: GET 2023-09-29T22:19:19.398 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.398 DEBUG:boto:Data: 2023-09-29T22:19:19.399 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.399 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.399 DEBUG:boto:Params: {} 2023-09-29T22:19:19.399 DEBUG:boto:Token: None 2023-09-29T22:19:19.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:19.399 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk= 2023-09-29T22:19:19.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qt4Pueg+obGcMJEEVKkGJ6Hr1Vk=', 'Content-Length': '0'} 2023-09-29T22:19:19.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046d8deee74c5cbb5-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.401 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.401 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.401 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.401 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.401 DEBUG:boto:Method: GET 2023-09-29T22:19:19.402 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.402 DEBUG:boto:Data: 2023-09-29T22:19:19.402 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.402 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.402 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.402 DEBUG:boto:Params: {} 2023-09-29T22:19:19.402 DEBUG:boto:Token: None 2023-09-29T22:19:19.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773bc2400baea90b-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.405 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.405 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.405 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.406 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.406 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.406 DEBUG:boto:Method: GET 2023-09-29T22:19:19.406 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:19.406 DEBUG:boto:Data: 2023-09-29T22:19:19.406 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.406 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.406 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.406 DEBUG:boto:Params: {} 2023-09-29T22:19:19.407 DEBUG:boto:Token: None 2023-09-29T22:19:19.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14a9620ea66122d-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.409 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:19.410 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.410 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.410 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.410 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.410 DEBUG:boto:Method: GET 2023-09-29T22:19:19.410 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.411 DEBUG:boto:Data: 2023-09-29T22:19:19.411 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.411 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.411 DEBUG:boto:Params: {} 2023-09-29T22:19:19.411 DEBUG:boto:Token: None 2023-09-29T22:19:19.411 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.411 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab48b80bb935d2b4-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.413 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.413 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.413 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.414 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.414 DEBUG:boto:Method: GET 2023-09-29T22:19:19.414 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.414 DEBUG:boto:Data: 2023-09-29T22:19:19.414 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.414 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.414 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.414 DEBUG:boto:Params: {} 2023-09-29T22:19:19.414 DEBUG:boto:Token: None 2023-09-29T22:19:19.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec49c9c68f98ab62-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.416 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.417 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.417 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.417 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.417 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.417 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.417 DEBUG:boto:Data: 2023-09-29T22:19:19.417 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.417 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.417 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.417 DEBUG:boto:Params: {} 2023-09-29T22:19:19.418 DEBUG:boto:Token: None 2023-09-29T22:19:19.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.418 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tlTQGS95br78RGS3cAGah6//asw= 2023-09-29T22:19:19.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tlTQGS95br78RGS3cAGah6//asw=', 'Content-Length': '0'} 2023-09-29T22:19:19.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000071dcf9e4fdc269e-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.420 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.420 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.420 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.420 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.420 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.420 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.420 DEBUG:boto:Data: 2023-09-29T22:19:19.421 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.421 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.421 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.421 DEBUG:boto:Params: {} 2023-09-29T22:19:19.421 DEBUG:boto:Token: None 2023-09-29T22:19:19.421 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tlTQGS95br78RGS3cAGah6//asw= 2023-09-29T22:19:19.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tlTQGS95br78RGS3cAGah6//asw=', 'Content-Length': '0'} 2023-09-29T22:19:19.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041822bd08c0da58f-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:19.423 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.423 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.423 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.423 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.424 DEBUG:boto:Method: GET 2023-09-29T22:19:19.424 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.424 DEBUG:boto:Data: 2023-09-29T22:19:19.424 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.424 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.424 DEBUG:boto:Params: {} 2023-09-29T22:19:19.424 DEBUG:boto:Token: None 2023-09-29T22:19:19.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.425 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fadc945163c2754-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.470 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.470 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.471 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.471 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.471 DEBUG:boto:Method: GET 2023-09-29T22:19:19.471 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.471 DEBUG:boto:Data: 2023-09-29T22:19:19.471 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.471 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.471 DEBUG:boto:Params: {} 2023-09-29T22:19:19.472 DEBUG:boto:Token: None 2023-09-29T22:19:19.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e917af80ffab7f84-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.474 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.474 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.474 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.474 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.475 DEBUG:boto:Method: GET 2023-09-29T22:19:19.475 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.475 DEBUG:boto:Data: 2023-09-29T22:19:19.475 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.475 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.475 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.475 DEBUG:boto:Params: {} 2023-09-29T22:19:19.475 DEBUG:boto:Token: None 2023-09-29T22:19:19.475 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9a76225519f618e-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.478 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.478 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.479 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.479 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.479 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.479 DEBUG:boto:Method: GET 2023-09-29T22:19:19.479 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.479 DEBUG:boto:Data: 2023-09-29T22:19:19.479 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.479 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.479 DEBUG:boto:Params: {} 2023-09-29T22:19:19.480 DEBUG:boto:Token: None 2023-09-29T22:19:19.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b1cf168061be5a-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.482 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.483 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.483 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.483 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.483 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.483 DEBUG:boto:Data: 2023-09-29T22:19:19.483 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.483 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.484 DEBUG:boto:Params: {} 2023-09-29T22:19:19.484 DEBUG:boto:Token: None 2023-09-29T22:19:19.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wbz+fV2nTzGLpbe5TEtm61BuIBg= 2023-09-29T22:19:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wbz+fV2nTzGLpbe5TEtm61BuIBg=', 'Content-Length': '0'} 2023-09-29T22:19:19.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5b19ad4c7953caa-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.486 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.486 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.487 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.487 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.487 DEBUG:boto:Data: 2023-09-29T22:19:19.487 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.487 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.487 DEBUG:boto:Params: {} 2023-09-29T22:19:19.487 DEBUG:boto:Token: None 2023-09-29T22:19:19.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wbz+fV2nTzGLpbe5TEtm61BuIBg= 2023-09-29T22:19:19.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wbz+fV2nTzGLpbe5TEtm61BuIBg=', 'Content-Length': '0'} 2023-09-29T22:19:19.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011a1f433f8c786c4-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:19.491 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.491 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.492 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.492 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.492 DEBUG:boto:Method: GET 2023-09-29T22:19:19.492 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.492 DEBUG:boto:Data: 2023-09-29T22:19:19.492 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.492 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.493 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.493 DEBUG:boto:Params: {} 2023-09-29T22:19:19.493 DEBUG:boto:Token: None 2023-09-29T22:19:19.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.493 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e7528e881cd9011-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.538 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:19.539 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.539 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.539 DEBUG:boto:Method: GET 2023-09-29T22:19:19.539 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.539 DEBUG:boto:Data: 2023-09-29T22:19:19.539 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.539 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.540 DEBUG:boto:Params: {} 2023-09-29T22:19:19.540 DEBUG:boto:Token: None 2023-09-29T22:19:19.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:19.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo= 2023-09-29T22:19:19.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CWyKoNTuQY37ZUDTF5y7kVV8Slo=', 'Content-Length': '0'} 2023-09-29T22:19:19.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034180c735d536c9e-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.543 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.543 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.543 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.543 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.543 DEBUG:boto:Method: GET 2023-09-29T22:19:19.543 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.543 DEBUG:boto:Data: 2023-09-29T22:19:19.543 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.543 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.543 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.544 DEBUG:boto:Params: {} 2023-09-29T22:19:19.544 DEBUG:boto:Token: None 2023-09-29T22:19:19.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c91ac16cd3c8dc-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.590 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.591 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.591 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.591 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.591 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.591 DEBUG:boto:Method: GET 2023-09-29T22:19:19.592 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:19.592 DEBUG:boto:Data: 2023-09-29T22:19:19.592 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.592 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.592 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.592 DEBUG:boto:Params: {} 2023-09-29T22:19:19.592 DEBUG:boto:Token: None 2023-09-29T22:19:19.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.593 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aef4225780415ae-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.595 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:19.596 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.596 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.596 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.596 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.596 DEBUG:boto:Method: GET 2023-09-29T22:19:19.597 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.597 DEBUG:boto:Data: 2023-09-29T22:19:19.597 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.597 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.597 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.597 DEBUG:boto:Params: {} 2023-09-29T22:19:19.598 DEBUG:boto:Token: None 2023-09-29T22:19:19.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020f8e293e3e39e40-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.600 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.600 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.600 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.600 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.601 DEBUG:boto:Method: GET 2023-09-29T22:19:19.601 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.601 DEBUG:boto:Data: 2023-09-29T22:19:19.601 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.601 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.601 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.601 DEBUG:boto:Params: {} 2023-09-29T22:19:19.602 DEBUG:boto:Token: None 2023-09-29T22:19:19.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d2d01f5171a328-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.646 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.647 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.647 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.648 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.648 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.648 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.648 DEBUG:boto:Data: 2023-09-29T22:19:19.649 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.649 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.649 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.649 DEBUG:boto:Params: {} 2023-09-29T22:19:19.650 DEBUG:boto:Token: None 2023-09-29T22:19:19.650 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.650 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:idXKrLonW7TyeaIJXlKZ+LiDOuw= 2023-09-29T22:19:19.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:idXKrLonW7TyeaIJXlKZ+LiDOuw=', 'Content-Length': '0'} 2023-09-29T22:19:19.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9183230e2b166ce-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.652 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.653 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.653 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.653 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.653 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.653 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.653 DEBUG:boto:Data: 2023-09-29T22:19:19.653 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.653 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.654 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.654 DEBUG:boto:Params: {} 2023-09-29T22:19:19.654 DEBUG:boto:Token: None 2023-09-29T22:19:19.654 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.654 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:idXKrLonW7TyeaIJXlKZ+LiDOuw= 2023-09-29T22:19:19.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:idXKrLonW7TyeaIJXlKZ+LiDOuw=', 'Content-Length': '0'} 2023-09-29T22:19:19.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002863c18638de2f75-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:19.657 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.657 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.657 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.657 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.657 DEBUG:boto:Method: GET 2023-09-29T22:19:19.657 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.657 DEBUG:boto:Data: 2023-09-29T22:19:19.658 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.658 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.658 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.658 DEBUG:boto:Params: {} 2023-09-29T22:19:19.658 DEBUG:boto:Token: None 2023-09-29T22:19:19.658 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.658 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d1e1d25139f5f54-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.702 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.703 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.703 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.703 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.703 DEBUG:boto:Method: GET 2023-09-29T22:19:19.703 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.703 DEBUG:boto:Data: 2023-09-29T22:19:19.703 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.703 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.703 DEBUG:boto:Params: {} 2023-09-29T22:19:19.704 DEBUG:boto:Token: None 2023-09-29T22:19:19.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f644af2055833dc-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.746 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.747 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.747 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.747 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.747 DEBUG:boto:Method: GET 2023-09-29T22:19:19.747 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.747 DEBUG:boto:Data: 2023-09-29T22:19:19.747 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.748 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.748 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.748 DEBUG:boto:Params: {} 2023-09-29T22:19:19.748 DEBUG:boto:Token: None 2023-09-29T22:19:19.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.749 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb7ce38f1aa1bbd-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.751 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.752 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.752 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.752 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.752 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.753 DEBUG:boto:Method: GET 2023-09-29T22:19:19.753 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.753 DEBUG:boto:Data: 2023-09-29T22:19:19.753 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.753 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.753 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.753 DEBUG:boto:Params: {} 2023-09-29T22:19:19.754 DEBUG:boto:Token: None 2023-09-29T22:19:19.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.754 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b33efc8b0f262f-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.757 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.757 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.757 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.757 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.757 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.757 DEBUG:boto:Data: 2023-09-29T22:19:19.758 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.758 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.758 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.758 DEBUG:boto:Params: {} 2023-09-29T22:19:19.758 DEBUG:boto:Token: None 2023-09-29T22:19:19.758 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.758 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJALrLPjrXiQDTvXR2iLtQItlQw= 2023-09-29T22:19:19.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJALrLPjrXiQDTvXR2iLtQItlQw=', 'Content-Length': '0'} 2023-09-29T22:19:19.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1795e67e867454b-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.761 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.761 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.761 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.762 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.762 DEBUG:boto:Data: 2023-09-29T22:19:19.762 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.762 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.763 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.763 DEBUG:boto:Params: {} 2023-09-29T22:19:19.763 DEBUG:boto:Token: None 2023-09-29T22:19:19.763 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.764 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJALrLPjrXiQDTvXR2iLtQItlQw= 2023-09-29T22:19:19.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJALrLPjrXiQDTvXR2iLtQItlQw=', 'Content-Length': '0'} 2023-09-29T22:19:19.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003cb48559ec6eec1-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:19.767 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.767 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.767 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.767 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.768 DEBUG:boto:Method: GET 2023-09-29T22:19:19.768 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.768 DEBUG:boto:Data: 2023-09-29T22:19:19.768 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.768 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.769 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.769 DEBUG:boto:Params: {} 2023-09-29T22:19:19.769 DEBUG:boto:Token: None 2023-09-29T22:19:19.769 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.770 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4ab0acc9ffb814d-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.814 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.815 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:19.815 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.815 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.815 DEBUG:boto:Method: GET 2023-09-29T22:19:19.815 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.816 DEBUG:boto:Data: 2023-09-29T22:19:19.816 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.816 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.816 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.816 DEBUG:boto:Params: {} 2023-09-29T22:19:19.817 DEBUG:boto:Token: None 2023-09-29T22:19:19.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:19.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U= 2023-09-29T22:19:19.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QNilU//U1Qrn40qlb1amMCYf90U=', 'Content-Length': '0'} 2023-09-29T22:19:19.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038099fec9f9efca3-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.862 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.863 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.863 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.863 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.863 DEBUG:boto:Method: GET 2023-09-29T22:19:19.863 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.863 DEBUG:boto:Data: 2023-09-29T22:19:19.863 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.864 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.864 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.864 DEBUG:boto:Params: {} 2023-09-29T22:19:19.864 DEBUG:boto:Token: None 2023-09-29T22:19:19.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e551b6f71a4af8d-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.867 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.867 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.868 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.868 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.868 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.868 DEBUG:boto:Method: GET 2023-09-29T22:19:19.868 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:19.869 DEBUG:boto:Data: 2023-09-29T22:19:19.869 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.869 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.869 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.869 DEBUG:boto:Params: {} 2023-09-29T22:19:19.869 DEBUG:boto:Token: None 2023-09-29T22:19:19.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7997f997ead27f9-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.914 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:19.915 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.916 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.916 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.916 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.916 DEBUG:boto:Method: GET 2023-09-29T22:19:19.916 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.916 DEBUG:boto:Data: 2023-09-29T22:19:19.916 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.916 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.916 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.917 DEBUG:boto:Params: {} 2023-09-29T22:19:19.917 DEBUG:boto:Token: None 2023-09-29T22:19:19.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.917 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039bbfb34e1e37ad6-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.920 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.920 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.920 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.920 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.920 DEBUG:boto:Method: GET 2023-09-29T22:19:19.920 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.920 DEBUG:boto:Data: 2023-09-29T22:19:19.920 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.921 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.921 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.921 DEBUG:boto:Params: {} 2023-09-29T22:19:19.921 DEBUG:boto:Token: None 2023-09-29T22:19:19.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.922 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006d20e56f60ef2cb-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.925 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.925 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.925 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.925 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.926 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.926 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.926 DEBUG:boto:Data: 2023-09-29T22:19:19.927 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.927 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.927 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.927 DEBUG:boto:Params: {} 2023-09-29T22:19:19.927 DEBUG:boto:Token: None 2023-09-29T22:19:19.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n/7K6hEu6STtmlcs75eCSOXZNb8= 2023-09-29T22:19:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n/7K6hEu6STtmlcs75eCSOXZNb8=', 'Content-Length': '0'} 2023-09-29T22:19:19.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5cdfcc575e7efdb-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.930 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.930 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.930 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.931 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.931 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.931 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.931 DEBUG:boto:Data: 2023-09-29T22:19:19.931 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.931 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.931 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.931 DEBUG:boto:Params: {} 2023-09-29T22:19:19.931 DEBUG:boto:Token: None 2023-09-29T22:19:19.931 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n/7K6hEu6STtmlcs75eCSOXZNb8= 2023-09-29T22:19:19.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n/7K6hEu6STtmlcs75eCSOXZNb8=', 'Content-Length': '0'} 2023-09-29T22:19:19.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b4e77636bc8c7b2-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:19.934 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.934 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.934 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.934 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.934 DEBUG:boto:Method: GET 2023-09-29T22:19:19.935 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.935 DEBUG:boto:Data: 2023-09-29T22:19:19.935 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.935 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.935 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.935 DEBUG:boto:Params: {} 2023-09-29T22:19:19.935 DEBUG:boto:Token: None 2023-09-29T22:19:19.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.935 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000809ee78dfe6ee885-0065174d67-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.938 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.938 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.938 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.938 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.938 DEBUG:boto:Method: GET 2023-09-29T22:19:19.938 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.939 DEBUG:boto:Data: 2023-09-29T22:19:19.939 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.939 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.939 DEBUG:boto:Params: {} 2023-09-29T22:19:19.939 DEBUG:boto:Token: None 2023-09-29T22:19:19.939 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:19.939 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI= 2023-09-29T22:19:19.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oFWKiOVJRa/MryxSWMKbNsFgqNI=', 'Content-Length': '0'} 2023-09-29T22:19:19.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000082f4de11286f015-0065174d67-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.986 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.987 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.987 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.987 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.987 DEBUG:boto:Method: GET 2023-09-29T22:19:19.987 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.987 DEBUG:boto:Data: 2023-09-29T22:19:19.987 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.988 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.988 DEBUG:boto:Params: {} 2023-09-29T22:19:19.988 DEBUG:boto:Token: None 2023-09-29T22:19:19.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eba7898559d0d4f2-0065174d67-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.992 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.992 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:19.992 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:19.992 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.992 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.993 DEBUG:boto:Method: GET 2023-09-29T22:19:19.993 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:19.993 DEBUG:boto:Data: 2023-09-29T22:19:19.993 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.993 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:19.993 DEBUG:boto:Port: 8000 2023-09-29T22:19:19.993 DEBUG:boto:Params: {} 2023-09-29T22:19:19.993 DEBUG:boto:Token: None 2023-09-29T22:19:19.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/ 2023-09-29T22:19:19.994 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk= 2023-09-29T22:19:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3CAuxMME/ixZS5IMDQqhM9Z2Itk=', 'Content-Length': '0'} 2023-09-29T22:19:19.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000281a59e813a0617e-0065174d67-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:19.996 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:19.997 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.997 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.997 DEBUG:boto:Method: HEAD 2023-09-29T22:19:19.997 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.997 DEBUG:boto:Data: 2023-09-29T22:19:19.998 DEBUG:boto:Headers: {} 2023-09-29T22:19:19.998 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:19.998 DEBUG:boto:Port: 8001 2023-09-29T22:19:19.998 DEBUG:boto:Params: {} 2023-09-29T22:19:19.998 DEBUG:boto:Token: None 2023-09-29T22:19:19.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:19 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:19.998 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs= 2023-09-29T22:19:19.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wyUO+9B+rI0oivQEp/3GxoiAdWs=', 'Content-Length': '0'} 2023-09-29T22:19:20.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066e8f6b075955252-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.001 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.001 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.001 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.001 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.001 DEBUG:boto:Data: 2023-09-29T22:19:20.001 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.001 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:20.002 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.002 DEBUG:boto:Params: {} 2023-09-29T22:19:20.002 DEBUG:boto:Token: None 2023-09-29T22:19:20.002 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.002 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jg0m9L70enk/3KmDpefzKQow6zI= 2023-09-29T22:19:20.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jg0m9L70enk/3KmDpefzKQow6zI=', 'Content-Length': '0'} 2023-09-29T22:19:20.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc49cb73d0748d91-0065174d68-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:20.004 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.005 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.005 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.005 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.005 DEBUG:boto:Method: GET 2023-09-29T22:19:20.005 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.005 DEBUG:boto:Data: 2023-09-29T22:19:20.006 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.006 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.006 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.006 DEBUG:boto:Params: {} 2023-09-29T22:19:20.006 DEBUG:boto:Token: None 2023-09-29T22:19:20.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d1336117c3a5e18-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.050 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.051 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.051 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.051 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.051 DEBUG:boto:Method: GET 2023-09-29T22:19:20.051 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.052 DEBUG:boto:Data: 2023-09-29T22:19:20.052 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.052 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:20.052 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.052 DEBUG:boto:Params: {} 2023-09-29T22:19:20.052 DEBUG:boto:Token: None 2023-09-29T22:19:20.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.053 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c7f9e42eae26406-0065174d68-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.102 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.103 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.103 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.103 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.103 DEBUG:boto:Method: GET 2023-09-29T22:19:20.104 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.104 DEBUG:boto:Data: 2023-09-29T22:19:20.104 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.104 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.104 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.105 DEBUG:boto:Params: {} 2023-09-29T22:19:20.105 DEBUG:boto:Token: None 2023-09-29T22:19:20.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.106 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9abcaad3486632-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.109 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.110 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.110 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.111 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.111 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.111 DEBUG:boto:Method: GET 2023-09-29T22:19:20.111 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.112 DEBUG:boto:Data: 2023-09-29T22:19:20.112 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.112 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:20.112 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.112 DEBUG:boto:Params: {} 2023-09-29T22:19:20.113 DEBUG:boto:Token: None 2023-09-29T22:19:20.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.113 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65fb61851dc46e7-0065174d68-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.162 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone3, test-zone1} 2023-09-29T22:19:20.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-16 zones={test-zone3, test-zone2} 2023-09-29T22:19:20.163 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.163 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.163 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.164 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:20.164 DEBUG:boto:Data: 2023-09-29T22:19:20.164 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.164 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.164 DEBUG:boto:Params: {} 2023-09-29T22:19:20.164 DEBUG:boto:Token: None 2023-09-29T22:19:20.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aMzUzt4KaIv+guWqnMuMZluF/9s= 2023-09-29T22:19:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aMzUzt4KaIv+guWqnMuMZluF/9s=', 'Content-Length': '0'} 2023-09-29T22:19:20.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e80acf849b5489f3-0065174d68-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.167 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.167 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.168 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.168 DEBUG:boto:Path: /dmuhsu-16/ 2023-09-29T22:19:20.168 DEBUG:boto:Data: 2023-09-29T22:19:20.168 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.168 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.168 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.168 DEBUG:boto:Params: {} 2023-09-29T22:19:20.168 DEBUG:boto:Token: None 2023-09-29T22:19:20.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aMzUzt4KaIv+guWqnMuMZluF/9s= 2023-09-29T22:19:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aMzUzt4KaIv+guWqnMuMZluF/9s=', 'Content-Length': '0'} 2023-09-29T22:19:20.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8c598a5e63ed2e2-0065174d68-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:20.172 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.172 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.172 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.172 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.172 DEBUG:boto:Method: GET 2023-09-29T22:19:20.172 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.172 DEBUG:boto:Data: 2023-09-29T22:19:20.172 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.173 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.173 DEBUG:boto:Params: {} 2023-09-29T22:19:20.173 DEBUG:boto:Token: None 2023-09-29T22:19:20.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/?versions 2023-09-29T22:19:20.173 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI= 2023-09-29T22:19:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI=', 'Content-Length': '0'} 2023-09-29T22:19:20.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce757b6d4c221a6a-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.176 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:20.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:20.177 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:20.177 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.177 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.177 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.177 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.177 DEBUG:boto:Method: GET 2023-09-29T22:19:20.177 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.178 DEBUG:boto:Data: 2023-09-29T22:19:20.178 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.178 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.178 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.178 DEBUG:boto:Params: {} 2023-09-29T22:19:20.178 DEBUG:boto:Token: None 2023-09-29T22:19:20.178 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/?versions 2023-09-29T22:19:20.178 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI= 2023-09-29T22:19:20.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI=', 'Content-Length': '0'} 2023-09-29T22:19:20.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0e0cc84a9e12cd-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.181 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:20.182 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.182 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.183 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.183 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.183 DEBUG:boto:Method: GET 2023-09-29T22:19:20.183 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.183 DEBUG:boto:Data: 2023-09-29T22:19:20.183 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.183 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.183 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.183 DEBUG:boto:Params: {} 2023-09-29T22:19:20.183 DEBUG:boto:Token: None 2023-09-29T22:19:20.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/?versions 2023-09-29T22:19:20.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI= 2023-09-29T22:19:20.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI=', 'Content-Length': '0'} 2023-09-29T22:19:20.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099bd54a5530d9194-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.186 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.187 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.187 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.187 DEBUG:boto:path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.187 DEBUG:boto:auth_path=/dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.187 DEBUG:boto:Method: GET 2023-09-29T22:19:20.187 DEBUG:boto:Path: /dmuhsu-16/?versions&max-keys=999 2023-09-29T22:19:20.188 DEBUG:boto:Data: 2023-09-29T22:19:20.188 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.188 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.188 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.188 DEBUG:boto:Params: {} 2023-09-29T22:19:20.188 DEBUG:boto:Token: None 2023-09-29T22:19:20.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/?versions 2023-09-29T22:19:20.188 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI= 2023-09-29T22:19:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtavMYG9V1iHorl4KPNjdnhToBI=', 'Content-Length': '0'} 2023-09-29T22:19:20.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f99a4eda1d193fd-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.191 DEBUG:boto:b'dmuhsu-16999falsedummyOIc2TWNpEVGciUsZx.KUd8STsRKRbeetrue2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1.KHBk1SFxtSzjOcXg76bpeg0aBZno-Htrue2023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcStrue2023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3F3EUPv9aCqBqw-KhN4eCQqgdKvZI4frtrue2023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:20.191 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.192 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.192 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.192 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.192 DEBUG:boto:Method: GET 2023-09-29T22:19:20.192 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.192 DEBUG:boto:Data: 2023-09-29T22:19:20.192 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.192 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.193 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.193 DEBUG:boto:Params: {} 2023-09-29T22:19:20.193 DEBUG:boto:Token: None 2023-09-29T22:19:20.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016a1bda8cecea6b7-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.195 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.195 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.195 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.195 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.196 DEBUG:boto:Method: GET 2023-09-29T22:19:20.196 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.196 DEBUG:boto:Data: 2023-09-29T22:19:20.196 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.196 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.196 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.196 DEBUG:boto:Params: {} 2023-09-29T22:19:20.196 DEBUG:boto:Token: None 2023-09-29T22:19:20.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f2d2c6d80cc37f6-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.199 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.199 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.199 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.199 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.199 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.200 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.200 DEBUG:boto:Data: 2023-09-29T22:19:20.200 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.200 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.200 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.200 DEBUG:boto:Params: {} 2023-09-29T22:19:20.200 DEBUG:boto:Token: None 2023-09-29T22:19:20.200 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hpt8HmxDznUteYgTdxGHuctEb48= 2023-09-29T22:19:20.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hpt8HmxDznUteYgTdxGHuctEb48=', 'Content-Length': '0'} 2023-09-29T22:19:20.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a197a0414cfbac3-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.203 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.203 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.203 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.203 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.203 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.203 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.203 DEBUG:boto:Data: 2023-09-29T22:19:20.203 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.204 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.204 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.204 DEBUG:boto:Params: {} 2023-09-29T22:19:20.204 DEBUG:boto:Token: None 2023-09-29T22:19:20.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hpt8HmxDznUteYgTdxGHuctEb48= 2023-09-29T22:19:20.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hpt8HmxDznUteYgTdxGHuctEb48=', 'Content-Length': '0'} 2023-09-29T22:19:20.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d3d351aa5df3efb-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:20.207 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.207 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.207 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.207 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.207 DEBUG:boto:Method: GET 2023-09-29T22:19:20.207 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.207 DEBUG:boto:Data: 2023-09-29T22:19:20.208 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.208 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.208 DEBUG:boto:Params: {} 2023-09-29T22:19:20.208 DEBUG:boto:Token: None 2023-09-29T22:19:20.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000501635bb83ebd371-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.210 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.210 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.210 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.211 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.211 DEBUG:boto:Method: GET 2023-09-29T22:19:20.211 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.211 DEBUG:boto:Data: 2023-09-29T22:19:20.211 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.211 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.211 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.211 DEBUG:boto:Params: {} 2023-09-29T22:19:20.211 DEBUG:boto:Token: None 2023-09-29T22:19:20.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071001071b2709175-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.214 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.214 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.214 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.214 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.214 DEBUG:boto:Method: GET 2023-09-29T22:19:20.214 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.214 DEBUG:boto:Data: 2023-09-29T22:19:20.214 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.215 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.215 DEBUG:boto:Params: {} 2023-09-29T22:19:20.215 DEBUG:boto:Token: None 2023-09-29T22:19:20.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.215 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d649d577b9aa82f-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.258 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.259 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.259 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.259 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.259 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.259 DEBUG:boto:Method: GET 2023-09-29T22:19:20.259 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.260 DEBUG:boto:Data: 2023-09-29T22:19:20.260 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.260 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.260 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.260 DEBUG:boto:Params: {} 2023-09-29T22:19:20.260 DEBUG:boto:Token: None 2023-09-29T22:19:20.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000200965329180bc41-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.263 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.264 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.264 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.264 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.264 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:20.264 DEBUG:boto:Data: 2023-09-29T22:19:20.264 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.264 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.264 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.264 DEBUG:boto:Params: {} 2023-09-29T22:19:20.265 DEBUG:boto:Token: None 2023-09-29T22:19:20.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy 2023-09-29T22:19:20.265 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:C3V0hi2OpHUnriKF1T8ordoNFno= 2023-09-29T22:19:20.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:C3V0hi2OpHUnriKF1T8ordoNFno=', 'Content-Length': '0'} 2023-09-29T22:19:20.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007977ade75a978041-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.267 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.268 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.268 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.268 DEBUG:boto:Path: /dmuhsu-16/dummy 2023-09-29T22:19:20.268 DEBUG:boto:Data: 2023-09-29T22:19:20.268 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.268 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.268 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.268 DEBUG:boto:Params: {} 2023-09-29T22:19:20.268 DEBUG:boto:Token: None 2023-09-29T22:19:20.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy 2023-09-29T22:19:20.269 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:C3V0hi2OpHUnriKF1T8ordoNFno= 2023-09-29T22:19:20.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:C3V0hi2OpHUnriKF1T8ordoNFno=', 'Content-Length': '0'} 2023-09-29T22:19:20.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000613fe29ccd6f42bb-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:20.271 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.272 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.272 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.272 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.272 DEBUG:boto:Method: GET 2023-09-29T22:19:20.272 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.272 DEBUG:boto:Data: 2023-09-29T22:19:20.272 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.272 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.272 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.273 DEBUG:boto:Params: {} 2023-09-29T22:19:20.273 DEBUG:boto:Token: None 2023-09-29T22:19:20.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.273 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be9d8de1b9287a0a-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.275 DEBUG:boto:path=/dmuhsu-16/dummy 2023-09-29T22:19:20.275 DEBUG:boto:auth_path=/dmuhsu-16/dummy 2023-09-29T22:19:20.275 DEBUG:boto:path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.275 DEBUG:boto:auth_path=/dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.275 DEBUG:boto:Method: GET 2023-09-29T22:19:20.275 DEBUG:boto:Path: /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.275 DEBUG:boto:Data: 2023-09-29T22:19:20.275 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.276 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.276 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.276 DEBUG:boto:Params: {} 2023-09-29T22:19:20.276 DEBUG:boto:Token: None 2023-09-29T22:19:20.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/dummy?versionId=OIc2TWNpEVGciUsZx.KUd8STsRKRbee 2023-09-29T22:19:20.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY= 2023-09-29T22:19:20.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tn7ntiX9iDzwJNuEiECe3nk7CZY=', 'Content-Length': '0'} 2023-09-29T22:19:20.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:25 GMT'), ('x-amz-version-id', 'OIc2TWNpEVGciUsZx.KUd8STsRKRbee'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099f0ffd75297a43d-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.278 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.278 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.278 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.279 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.279 DEBUG:boto:Method: GET 2023-09-29T22:19:20.279 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.279 DEBUG:boto:Data: 2023-09-29T22:19:20.279 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.279 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.279 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.279 DEBUG:boto:Params: {} 2023-09-29T22:19:20.281 DEBUG:boto:Token: None 2023-09-29T22:19:20.281 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.282 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a015b17960cefb-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.330 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.331 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.331 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.331 DEBUG:boto:path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.331 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.331 DEBUG:boto:Method: GET 2023-09-29T22:19:20.331 DEBUG:boto:Path: /dmuhsu-16/?prefix=dummy 2023-09-29T22:19:20.332 DEBUG:boto:Data: 2023-09-29T22:19:20.332 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.332 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.332 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.332 DEBUG:boto:Params: {} 2023-09-29T22:19:20.332 DEBUG:boto:Token: None 2023-09-29T22:19:20.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b77e83acc7fafe1-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.336 DEBUG:boto:b'dmuhsu-16dummy1000falsedummy2023-09-29T22:15:25.517Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:20.336 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.336 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.336 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.337 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.337 DEBUG:boto:Method: GET 2023-09-29T22:19:20.337 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.337 DEBUG:boto:Data: 2023-09-29T22:19:20.337 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.337 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.337 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.337 DEBUG:boto:Params: {} 2023-09-29T22:19:20.338 DEBUG:boto:Token: None 2023-09-29T22:19:20.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079cd8f71e5d49222-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.340 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.341 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.341 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.341 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.341 DEBUG:boto:Method: GET 2023-09-29T22:19:20.341 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.341 DEBUG:boto:Data: 2023-09-29T22:19:20.342 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.342 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.342 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.342 DEBUG:boto:Params: {} 2023-09-29T22:19:20.342 DEBUG:boto:Token: None 2023-09-29T22:19:20.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056c15eb7875ea720-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.386 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.387 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.387 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.387 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.387 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.387 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.387 DEBUG:boto:Data: 2023-09-29T22:19:20.387 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.387 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.388 DEBUG:boto:Params: {} 2023-09-29T22:19:20.388 DEBUG:boto:Token: None 2023-09-29T22:19:20.388 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:daSiucAtq1MGo+JdktZUspTZHEU= 2023-09-29T22:19:20.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:daSiucAtq1MGo+JdktZUspTZHEU=', 'Content-Length': '0'} 2023-09-29T22:19:20.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6afddef4d6a7e4f-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.390 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.390 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.390 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.391 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.391 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.391 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.391 DEBUG:boto:Data: 2023-09-29T22:19:20.391 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.391 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.391 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.391 DEBUG:boto:Params: {} 2023-09-29T22:19:20.391 DEBUG:boto:Token: None 2023-09-29T22:19:20.392 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:daSiucAtq1MGo+JdktZUspTZHEU= 2023-09-29T22:19:20.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:daSiucAtq1MGo+JdktZUspTZHEU=', 'Content-Length': '0'} 2023-09-29T22:19:20.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066822e1abf3a948c-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:20.394 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.394 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.394 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.394 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.394 DEBUG:boto:Method: GET 2023-09-29T22:19:20.394 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.394 DEBUG:boto:Data: 2023-09-29T22:19:20.395 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.395 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.395 DEBUG:boto:Params: {} 2023-09-29T22:19:20.395 DEBUG:boto:Token: None 2023-09-29T22:19:20.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec72e7c1a5d2c71a-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.442 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.443 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.443 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.443 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.443 DEBUG:boto:Method: GET 2023-09-29T22:19:20.443 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.443 DEBUG:boto:Data: 2023-09-29T22:19:20.443 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.443 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.443 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.443 DEBUG:boto:Params: {} 2023-09-29T22:19:20.444 DEBUG:boto:Token: None 2023-09-29T22:19:20.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.444 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000311304616abdde51-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.486 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.487 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.487 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.487 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.487 DEBUG:boto:Method: GET 2023-09-29T22:19:20.487 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.487 DEBUG:boto:Data: 2023-09-29T22:19:20.487 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.488 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.488 DEBUG:boto:Params: {} 2023-09-29T22:19:20.488 DEBUG:boto:Token: None 2023-09-29T22:19:20.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe00bf27ce0c903d-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.491 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.491 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.492 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.492 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.492 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.492 DEBUG:boto:Method: GET 2023-09-29T22:19:20.492 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.492 DEBUG:boto:Data: 2023-09-29T22:19:20.492 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.492 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.492 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.493 DEBUG:boto:Params: {} 2023-09-29T22:19:20.493 DEBUG:boto:Token: None 2023-09-29T22:19:20.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.493 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e9e2870c86aeb50-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.496 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.496 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.496 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.497 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.497 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.497 DEBUG:boto:Data: 2023-09-29T22:19:20.497 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.497 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.497 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.497 DEBUG:boto:Params: {} 2023-09-29T22:19:20.497 DEBUG:boto:Token: None 2023-09-29T22:19:20.497 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.498 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cpuDvo9WMVNKWo+xLUq3GvYEiMs= 2023-09-29T22:19:20.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cpuDvo9WMVNKWo+xLUq3GvYEiMs=', 'Content-Length': '0'} 2023-09-29T22:19:20.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a47039fcf6652d-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.500 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.500 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.500 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.500 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.500 DEBUG:boto:Data: 2023-09-29T22:19:20.501 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.501 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.501 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.501 DEBUG:boto:Params: {} 2023-09-29T22:19:20.501 DEBUG:boto:Token: None 2023-09-29T22:19:20.501 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.501 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cpuDvo9WMVNKWo+xLUq3GvYEiMs= 2023-09-29T22:19:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cpuDvo9WMVNKWo+xLUq3GvYEiMs=', 'Content-Length': '0'} 2023-09-29T22:19:20.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082c9383045ed1f91-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:20.504 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.504 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.504 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.504 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.504 DEBUG:boto:Method: GET 2023-09-29T22:19:20.504 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.504 DEBUG:boto:Data: 2023-09-29T22:19:20.505 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.505 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.505 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.505 DEBUG:boto:Params: {} 2023-09-29T22:19:20.505 DEBUG:boto:Token: None 2023-09-29T22:19:20.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bbfb1259d121249-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.550 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.551 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1 2023-09-29T22:19:20.551 DEBUG:boto:path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.551 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.551 DEBUG:boto:Method: GET 2023-09-29T22:19:20.551 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.551 DEBUG:boto:Data: 2023-09-29T22:19:20.551 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.551 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.551 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.552 DEBUG:boto:Params: {} 2023-09-29T22:19:20.552 DEBUG:boto:Token: None 2023-09-29T22:19:20.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone1?versionId=.KHBk1SFxtSzjOcXg76bpeg0aBZno-H 2023-09-29T22:19:20.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0= 2023-09-29T22:19:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WrhGurHteI/g+k5Z9dSW/qIL4A0=', 'Content-Length': '0'} 2023-09-29T22:19:20.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.KHBk1SFxtSzjOcXg76bpeg0aBZno-H'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0a0ccdc5752cf9a-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.598 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.599 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.599 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.599 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.599 DEBUG:boto:Method: GET 2023-09-29T22:19:20.599 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.599 DEBUG:boto:Data: 2023-09-29T22:19:20.599 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.599 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.600 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.600 DEBUG:boto:Params: {} 2023-09-29T22:19:20.600 DEBUG:boto:Token: None 2023-09-29T22:19:20.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.600 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000082488ff0caa60b6-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.603 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.603 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.603 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.603 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.603 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.603 DEBUG:boto:Method: GET 2023-09-29T22:19:20.604 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone1 2023-09-29T22:19:20.604 DEBUG:boto:Data: 2023-09-29T22:19:20.604 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.604 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.604 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.604 DEBUG:boto:Params: {} 2023-09-29T22:19:20.604 DEBUG:boto:Token: None 2023-09-29T22:19:20.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753a659344dd97a0-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.607 DEBUG:boto:b'dmuhsu-16obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.225Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:20.607 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.607 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.608 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.608 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.608 DEBUG:boto:Method: GET 2023-09-29T22:19:20.608 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.608 DEBUG:boto:Data: 2023-09-29T22:19:20.608 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.608 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.609 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.609 DEBUG:boto:Params: {} 2023-09-29T22:19:20.609 DEBUG:boto:Token: None 2023-09-29T22:19:20.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b413dad5b6bb89c3-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.612 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.612 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.612 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.613 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.613 DEBUG:boto:Method: GET 2023-09-29T22:19:20.613 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.613 DEBUG:boto:Data: 2023-09-29T22:19:20.613 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.613 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.614 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.614 DEBUG:boto:Params: {} 2023-09-29T22:19:20.614 DEBUG:boto:Token: None 2023-09-29T22:19:20.614 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.614 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a51ad7f97649d12a-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.617 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.617 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.618 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.618 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.618 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.618 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.619 DEBUG:boto:Data: 2023-09-29T22:19:20.619 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.619 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.619 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.619 DEBUG:boto:Params: {} 2023-09-29T22:19:20.620 DEBUG:boto:Token: None 2023-09-29T22:19:20.620 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8OYcyUytOEzBL90jRKqMpZ6Ziko= 2023-09-29T22:19:20.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8OYcyUytOEzBL90jRKqMpZ6Ziko=', 'Content-Length': '0'} 2023-09-29T22:19:20.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024c28857ac00d8cb-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.623 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.623 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.623 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.623 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.623 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.623 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.624 DEBUG:boto:Data: 2023-09-29T22:19:20.624 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.624 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.624 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.625 DEBUG:boto:Params: {} 2023-09-29T22:19:20.625 DEBUG:boto:Token: None 2023-09-29T22:19:20.625 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8OYcyUytOEzBL90jRKqMpZ6Ziko= 2023-09-29T22:19:20.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8OYcyUytOEzBL90jRKqMpZ6Ziko=', 'Content-Length': '0'} 2023-09-29T22:19:20.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000015d9397c7937357-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:20.628 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.628 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.629 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.629 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.629 DEBUG:boto:Method: GET 2023-09-29T22:19:20.629 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.630 DEBUG:boto:Data: 2023-09-29T22:19:20.630 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.630 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.630 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.630 DEBUG:boto:Params: {} 2023-09-29T22:19:20.630 DEBUG:boto:Token: None 2023-09-29T22:19:20.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.631 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5187534fd0f5afa-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.674 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.675 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.675 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.675 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.675 DEBUG:boto:Method: GET 2023-09-29T22:19:20.675 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.675 DEBUG:boto:Data: 2023-09-29T22:19:20.675 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.675 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.676 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.676 DEBUG:boto:Params: {} 2023-09-29T22:19:20.676 DEBUG:boto:Token: None 2023-09-29T22:19:20.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad17e71cc42f2279-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.679 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.679 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.679 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.679 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.679 DEBUG:boto:Method: GET 2023-09-29T22:19:20.679 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.680 DEBUG:boto:Data: 2023-09-29T22:19:20.680 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.680 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.680 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.680 DEBUG:boto:Params: {} 2023-09-29T22:19:20.680 DEBUG:boto:Token: None 2023-09-29T22:19:20.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b67091325700515-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.683 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.684 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.684 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.684 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.684 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.684 DEBUG:boto:Method: GET 2023-09-29T22:19:20.685 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.685 DEBUG:boto:Data: 2023-09-29T22:19:20.685 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.685 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.685 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.686 DEBUG:boto:Params: {} 2023-09-29T22:19:20.686 DEBUG:boto:Token: None 2023-09-29T22:19:20.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033abd1715bd8579b-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.734 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.735 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.735 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.736 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.736 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.736 DEBUG:boto:Data: 2023-09-29T22:19:20.736 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.737 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.737 DEBUG:boto:Params: {} 2023-09-29T22:19:20.737 DEBUG:boto:Token: None 2023-09-29T22:19:20.737 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.738 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZZOT3Pt/du20PtnZlhN9yZa9FRU= 2023-09-29T22:19:20.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZZOT3Pt/du20PtnZlhN9yZa9FRU=', 'Content-Length': '0'} 2023-09-29T22:19:20.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7b6407d3db25af1-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.740 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.741 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.741 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.741 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.741 DEBUG:boto:Data: 2023-09-29T22:19:20.741 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.742 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.742 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.742 DEBUG:boto:Params: {} 2023-09-29T22:19:20.742 DEBUG:boto:Token: None 2023-09-29T22:19:20.742 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.743 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZZOT3Pt/du20PtnZlhN9yZa9FRU= 2023-09-29T22:19:20.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZZOT3Pt/du20PtnZlhN9yZa9FRU=', 'Content-Length': '0'} 2023-09-29T22:19:20.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083d1281e21ff456c-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:20.746 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.746 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.746 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.746 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.746 DEBUG:boto:Method: GET 2023-09-29T22:19:20.747 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.747 DEBUG:boto:Data: 2023-09-29T22:19:20.747 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.747 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.747 DEBUG:boto:Params: {} 2023-09-29T22:19:20.748 DEBUG:boto:Token: None 2023-09-29T22:19:20.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c67d675c0c2750ae-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.794 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.795 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2 2023-09-29T22:19:20.795 DEBUG:boto:path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.795 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.796 DEBUG:boto:Method: GET 2023-09-29T22:19:20.796 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.796 DEBUG:boto:Data: 2023-09-29T22:19:20.796 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.797 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.797 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.797 DEBUG:boto:Params: {} 2023-09-29T22:19:20.797 DEBUG:boto:Token: None 2023-09-29T22:19:20.798 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone2?versionId=.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS 2023-09-29T22:19:20.798 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE= 2023-09-29T22:19:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RjXsh/zwSdb5GbymazWQX7BgpiE=', 'Content-Length': '0'} 2023-09-29T22:19:20.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '.RF2Z7Qnze8zGcOwzVq0gzDxIv.JVcS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4735d1531fdaf00-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.801 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.801 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.801 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.802 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.802 DEBUG:boto:Method: GET 2023-09-29T22:19:20.802 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.802 DEBUG:boto:Data: 2023-09-29T22:19:20.802 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.802 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.803 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.803 DEBUG:boto:Params: {} 2023-09-29T22:19:20.803 DEBUG:boto:Token: None 2023-09-29T22:19:20.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.803 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4d32b17b41c806-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.806 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.807 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.807 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.807 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.807 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.808 DEBUG:boto:Method: GET 2023-09-29T22:19:20.808 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone2 2023-09-29T22:19:20.808 DEBUG:boto:Data: 2023-09-29T22:19:20.808 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.808 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.808 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.808 DEBUG:boto:Params: {} 2023-09-29T22:19:20.809 DEBUG:boto:Token: None 2023-09-29T22:19:20.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:20.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000790199504bec4722-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.812 DEBUG:boto:b'dmuhsu-16obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.321Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:20.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:20.813 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.813 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.813 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.814 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.814 DEBUG:boto:Method: GET 2023-09-29T22:19:20.814 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.814 DEBUG:boto:Data: 2023-09-29T22:19:20.815 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.815 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.815 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.815 DEBUG:boto:Params: {} 2023-09-29T22:19:20.816 DEBUG:boto:Token: None 2023-09-29T22:19:20.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ecb8ccd2fe44d8d-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.819 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.819 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.820 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.820 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.820 DEBUG:boto:Method: GET 2023-09-29T22:19:20.820 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.820 DEBUG:boto:Data: 2023-09-29T22:19:20.821 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.821 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.821 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.821 DEBUG:boto:Params: {} 2023-09-29T22:19:20.821 DEBUG:boto:Token: None 2023-09-29T22:19:20.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.822 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0c2438380b7979a-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.824 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.825 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.825 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.825 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.826 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.826 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.826 DEBUG:boto:Data: 2023-09-29T22:19:20.827 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.827 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.828 DEBUG:boto:Params: {} 2023-09-29T22:19:20.828 DEBUG:boto:Token: None 2023-09-29T22:19:20.828 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKFj0WtmmtdQAqnc4ZSVLTWGTgA= 2023-09-29T22:19:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKFj0WtmmtdQAqnc4ZSVLTWGTgA=', 'Content-Length': '0'} 2023-09-29T22:19:20.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e650af4f5d0bd836-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.832 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.832 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.832 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.833 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.833 DEBUG:boto:Method: HEAD 2023-09-29T22:19:20.833 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.834 DEBUG:boto:Data: 2023-09-29T22:19:20.834 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.834 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.835 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.835 DEBUG:boto:Params: {} 2023-09-29T22:19:20.836 DEBUG:boto:Token: None 2023-09-29T22:19:20.836 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKFj0WtmmtdQAqnc4ZSVLTWGTgA= 2023-09-29T22:19:20.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKFj0WtmmtdQAqnc4ZSVLTWGTgA=', 'Content-Length': '0'} 2023-09-29T22:19:20.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000243742db3f300cbf-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:20.840 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.840 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.840 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.840 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.841 DEBUG:boto:Method: GET 2023-09-29T22:19:20.841 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.841 DEBUG:boto:Data: 2023-09-29T22:19:20.841 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.842 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.842 DEBUG:boto:Params: {} 2023-09-29T22:19:20.842 DEBUG:boto:Token: None 2023-09-29T22:19:20.842 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f456b2427e4b5b7c-0065174d68-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.886 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.946 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:20.946 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.946 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.946 DEBUG:boto:Method: GET 2023-09-29T22:19:20.946 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.947 DEBUG:boto:Data: 2023-09-29T22:19:20.947 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.947 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:20.947 DEBUG:boto:Port: 8000 2023-09-29T22:19:20.947 DEBUG:boto:Params: {} 2023-09-29T22:19:20.947 DEBUG:boto:Token: None 2023-09-29T22:19:20.947 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:20.947 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8= 2023-09-29T22:19:20.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okC1QCMU7Do4jRdnHO20qyln0b8=', 'Content-Length': '0'} 2023-09-29T22:19:20.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037dd7cc67ce0617a-0065174d68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:20.995 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:20.995 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:20.995 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.995 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.995 DEBUG:boto:Method: GET 2023-09-29T22:19:20.995 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:20.996 DEBUG:boto:Data: 2023-09-29T22:19:20.996 DEBUG:boto:Headers: {} 2023-09-29T22:19:20.996 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:20.996 DEBUG:boto:Port: 8001 2023-09-29T22:19:20.996 DEBUG:boto:Params: {} 2023-09-29T22:19:20.996 DEBUG:boto:Token: None 2023-09-29T22:19:20.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:20 GMT /dmuhsu-16/ 2023-09-29T22:19:20.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ= 2023-09-29T22:19:20.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:20 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NI0w33GxJi3PKeMWa2TlVFvygDQ=', 'Content-Length': '0'} 2023-09-29T22:19:21.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e37e2805510a1e2-0065174d68-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.000 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.000 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:21.001 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:21.001 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.001 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.001 DEBUG:boto:Method: GET 2023-09-29T22:19:21.001 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.001 DEBUG:boto:Data: 2023-09-29T22:19:21.001 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.001 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.002 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.002 DEBUG:boto:Params: {} 2023-09-29T22:19:21.002 DEBUG:boto:Token: None 2023-09-29T22:19:21.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/ 2023-09-29T22:19:21.002 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg= 2023-09-29T22:19:21.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg=', 'Content-Length': '0'} 2023-09-29T22:19:21.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e875816ad334a8dc-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.050 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.051 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.051 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.051 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.051 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.051 DEBUG:boto:Data: 2023-09-29T22:19:21.051 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.052 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:21.052 DEBUG:boto:Port: 8001 2023-09-29T22:19:21.052 DEBUG:boto:Params: {} 2023-09-29T22:19:21.052 DEBUG:boto:Token: None 2023-09-29T22:19:21.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+b1S4O7jvJEm1gxtAN/TLLU1H84= 2023-09-29T22:19:21.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+b1S4O7jvJEm1gxtAN/TLLU1H84=', 'Content-Length': '0'} 2023-09-29T22:19:21.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003147b2646f03b465-0065174d69-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.055 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.055 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.055 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.055 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.055 DEBUG:boto:Data: 2023-09-29T22:19:21.055 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.055 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.056 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.056 DEBUG:boto:Params: {} 2023-09-29T22:19:21.056 DEBUG:boto:Token: None 2023-09-29T22:19:21.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+b1S4O7jvJEm1gxtAN/TLLU1H84= 2023-09-29T22:19:21.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+b1S4O7jvJEm1gxtAN/TLLU1H84=', 'Content-Length': '0'} 2023-09-29T22:19:21.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008da782cc2a5c8fcf-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:21.059 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.059 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.059 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.059 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.059 DEBUG:boto:Method: GET 2023-09-29T22:19:21.059 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.059 DEBUG:boto:Data: 2023-09-29T22:19:21.060 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:21.060 DEBUG:boto:Port: 8001 2023-09-29T22:19:21.060 DEBUG:boto:Params: {} 2023-09-29T22:19:21.060 DEBUG:boto:Token: None 2023-09-29T22:19:21.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:opDrgFOF0ck2uYpk/9bMkIFj2OQ= 2023-09-29T22:19:21.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:opDrgFOF0ck2uYpk/9bMkIFj2OQ=', 'Content-Length': '0'} 2023-09-29T22:19:21.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000548cedb50e61b2a4-0065174d69-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.062 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.062 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3 2023-09-29T22:19:21.063 DEBUG:boto:path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.063 DEBUG:boto:auth_path=/dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.063 DEBUG:boto:Method: GET 2023-09-29T22:19:21.063 DEBUG:boto:Path: /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.063 DEBUG:boto:Data: 2023-09-29T22:19:21.063 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.063 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.063 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.063 DEBUG:boto:Params: {} 2023-09-29T22:19:21.064 DEBUG:boto:Token: None 2023-09-29T22:19:21.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/obj-test-zone3?versionId=F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr 2023-09-29T22:19:21.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:opDrgFOF0ck2uYpk/9bMkIFj2OQ= 2023-09-29T22:19:21.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:opDrgFOF0ck2uYpk/9bMkIFj2OQ=', 'Content-Length': '0'} 2023-09-29T22:19:21.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'F3EUPv9aCqBqw-KhN4eCQqgdKvZI4fr'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f369b6b8751306a5-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.110 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:21.111 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:21.111 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.111 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.111 DEBUG:boto:Method: GET 2023-09-29T22:19:21.111 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.111 DEBUG:boto:Data: 2023-09-29T22:19:21.111 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:21.111 DEBUG:boto:Port: 8001 2023-09-29T22:19:21.112 DEBUG:boto:Params: {} 2023-09-29T22:19:21.112 DEBUG:boto:Token: None 2023-09-29T22:19:21.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/ 2023-09-29T22:19:21.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg= 2023-09-29T22:19:21.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg=', 'Content-Length': '0'} 2023-09-29T22:19:21.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023a46355ec5a1f11-0065174d69-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.115 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.115 DEBUG:boto:path=/dmuhsu-16/ 2023-09-29T22:19:21.116 DEBUG:boto:auth_path=/dmuhsu-16/ 2023-09-29T22:19:21.116 DEBUG:boto:path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.116 DEBUG:boto:auth_path=/dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.116 DEBUG:boto:Method: GET 2023-09-29T22:19:21.116 DEBUG:boto:Path: /dmuhsu-16/?prefix=obj-test-zone3 2023-09-29T22:19:21.116 DEBUG:boto:Data: 2023-09-29T22:19:21.116 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.116 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.116 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.117 DEBUG:boto:Params: {} 2023-09-29T22:19:21.117 DEBUG:boto:Token: None 2023-09-29T22:19:21.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-16/ 2023-09-29T22:19:21.117 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg= 2023-09-29T22:19:21.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BFITeecq6BF9kbvmukhtuhQYNhg=', 'Content-Length': '0'} 2023-09-29T22:19:21.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c4481b8da30e1c6-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.162 DEBUG:boto:b'dmuhsu-16obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.417Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-16 zones={test-zone3, test-zone2} 2023-09-29T22:19:21.163 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-17 2023-09-29T22:19:21.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.165 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.224 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.224 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.260 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:21.260+0000 7f2b18d4b880 1 bucket sync caught up with source: 2023-09-29T22:19:21.260 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.46.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:19:21.261 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.46.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:19:21.261 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:21.260+0000 7f2b18d4b880 0 bucket checkpoint complete 2023-09-29T22:19:21.274 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-17 2023-09-29T22:19:21.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.275 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.334 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.372+0000 7f5336644880 1 bucket sync caught up with source: 2023-09-29T22:19:21.372 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.46.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:19:21.372 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.46.6, , 00000000003.9.10, , 00000000007.17.10, , , , , ] 2023-09-29T22:19:21.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.372+0000 7f5336644880 0 bucket checkpoint complete 2023-09-29T22:19:21.384 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-17 2023-09-29T22:19:21.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.384 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.438 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.438 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.472+0000 7f43c331e880 1 bucket sync caught up with source: 2023-09-29T22:19:21.472 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.27.10, , 00000000011.39.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:19:21.472 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.27.10, , 00000000011.39.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:19:21.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.472+0000 7f43c331e880 0 bucket checkpoint complete 2023-09-29T22:19:21.484 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-17 2023-09-29T22:19:21.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.484 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.535 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.568+0000 7ff248811880 1 bucket sync caught up with source: 2023-09-29T22:19:21.569 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.27.10, , 00000000011.39.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:19:21.569 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.27.10, , 00000000011.39.6, , 00000000008.37.10, , , , , ] 2023-09-29T22:19:21.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.568+0000 7ff248811880 0 bucket checkpoint complete 2023-09-29T22:19:21.580 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-17 2023-09-29T22:19:21.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.581 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.634 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.634 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.668+0000 7f1c65923880 1 bucket sync caught up with source: 2023-09-29T22:19:21.669 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000003.21.10, , 00000000015.52.6, , , , , ] 2023-09-29T22:19:21.669 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000003.21.10, , 00000000015.52.6, , , , , ] 2023-09-29T22:19:21.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:21.668+0000 7f1c65923880 0 bucket checkpoint complete 2023-09-29T22:19:21.680 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-17 2023-09-29T22:19:21.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:21.681 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:21.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:21.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:21.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:21.768+0000 7fd1ec244880 1 bucket sync caught up with source: 2023-09-29T22:19:21.768 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.29.10, , 00000000003.21.10, , 00000000015.52.6, , , , , ] 2023-09-29T22:19:21.768 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.29.10, , 00000000003.21.10, , 00000000015.52.6, , , , , ] 2023-09-29T22:19:21.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:21.768+0000 7fd1ec244880 0 bucket checkpoint complete 2023-09-29T22:19:21.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone2, test-zone1} 2023-09-29T22:19:21.781 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.781 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.782 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.782 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:21.782 DEBUG:boto:Data: 2023-09-29T22:19:21.782 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.782 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.783 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.783 DEBUG:boto:Params: {} 2023-09-29T22:19:21.783 DEBUG:boto:Token: None 2023-09-29T22:19:21.783 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.783 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AzprIUxC1HHatkkSruQwXgPrN3w= 2023-09-29T22:19:21.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AzprIUxC1HHatkkSruQwXgPrN3w=', 'Content-Length': '0'} 2023-09-29T22:19:21.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c05fec9cb1d1d08d-0065174d69-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.786 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.786 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.786 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.786 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:21.787 DEBUG:boto:Data: 2023-09-29T22:19:21.787 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.787 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.787 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.787 DEBUG:boto:Params: {} 2023-09-29T22:19:21.787 DEBUG:boto:Token: None 2023-09-29T22:19:21.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AzprIUxC1HHatkkSruQwXgPrN3w= 2023-09-29T22:19:21.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AzprIUxC1HHatkkSruQwXgPrN3w=', 'Content-Length': '0'} 2023-09-29T22:19:21.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055436e83270c58eb-0065174d69-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.791 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:21.791 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.791 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.791 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.791 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.792 DEBUG:boto:Method: GET 2023-09-29T22:19:21.792 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.792 DEBUG:boto:Data: 2023-09-29T22:19:21.792 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.792 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.792 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.792 DEBUG:boto:Params: {} 2023-09-29T22:19:21.792 DEBUG:boto:Token: None 2023-09-29T22:19:21.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/?versions 2023-09-29T22:19:21.793 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc= 2023-09-29T22:19:21.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc=', 'Content-Length': '0'} 2023-09-29T22:19:21.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad32c6e30c766576-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.796 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.796 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:21.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:21.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:21.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:21.797 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:21.797 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.797 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.797 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.798 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.798 DEBUG:boto:Method: GET 2023-09-29T22:19:21.798 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.798 DEBUG:boto:Data: 2023-09-29T22:19:21.798 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.799 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.799 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.799 DEBUG:boto:Params: {} 2023-09-29T22:19:21.799 DEBUG:boto:Token: None 2023-09-29T22:19:21.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/?versions 2023-09-29T22:19:21.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc= 2023-09-29T22:19:21.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc=', 'Content-Length': '0'} 2023-09-29T22:19:21.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065fa9bcf382da7ae-0065174d69-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.803 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.804 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:21.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:21.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:21.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:21.805 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.805 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.805 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.805 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.805 DEBUG:boto:Method: GET 2023-09-29T22:19:21.806 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.806 DEBUG:boto:Data: 2023-09-29T22:19:21.806 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.806 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.806 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.806 DEBUG:boto:Params: {} 2023-09-29T22:19:21.807 DEBUG:boto:Token: None 2023-09-29T22:19:21.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/?versions 2023-09-29T22:19:21.807 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc= 2023-09-29T22:19:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc=', 'Content-Length': '0'} 2023-09-29T22:19:21.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a56e8f3166d6f1-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.810 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.810 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.811 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.811 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.811 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.811 DEBUG:boto:Method: GET 2023-09-29T22:19:21.811 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:21.811 DEBUG:boto:Data: 2023-09-29T22:19:21.811 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.812 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.812 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.812 DEBUG:boto:Params: {} 2023-09-29T22:19:21.812 DEBUG:boto:Token: None 2023-09-29T22:19:21.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/?versions 2023-09-29T22:19:21.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc= 2023-09-29T22:19:21.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqLBoECwr4s2oE++mOrB+N0CuPc=', 'Content-Length': '0'} 2023-09-29T22:19:21.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca77b04045cd12aa-0065174d69-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.815 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:21.815 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.815 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.816 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.816 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.816 DEBUG:boto:Method: GET 2023-09-29T22:19:21.816 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.816 DEBUG:boto:Data: 2023-09-29T22:19:21.816 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.816 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.816 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.817 DEBUG:boto:Params: {} 2023-09-29T22:19:21.817 DEBUG:boto:Token: None 2023-09-29T22:19:21.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000daf4ada1ff9787e8-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.819 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.819 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.819 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.820 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.820 DEBUG:boto:Method: GET 2023-09-29T22:19:21.820 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.820 DEBUG:boto:Data: 2023-09-29T22:19:21.821 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.821 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.821 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.821 DEBUG:boto:Params: {} 2023-09-29T22:19:21.822 DEBUG:boto:Token: None 2023-09-29T22:19:21.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.822 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c22b06cc24edb29f-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.825 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.825 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.825 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.825 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.826 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.826 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.826 DEBUG:boto:Data: 2023-09-29T22:19:21.826 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.826 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.826 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.827 DEBUG:boto:Params: {} 2023-09-29T22:19:21.827 DEBUG:boto:Token: None 2023-09-29T22:19:21.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bcXQkYJDk1W2Asza5CyafSl8uxU= 2023-09-29T22:19:21.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bcXQkYJDk1W2Asza5CyafSl8uxU=', 'Content-Length': '0'} 2023-09-29T22:19:21.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0449a15eeb03c4b-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.829 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.830 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.830 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.830 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.830 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.830 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.830 DEBUG:boto:Data: 2023-09-29T22:19:21.831 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.831 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.831 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.831 DEBUG:boto:Params: {} 2023-09-29T22:19:21.831 DEBUG:boto:Token: None 2023-09-29T22:19:21.831 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bcXQkYJDk1W2Asza5CyafSl8uxU= 2023-09-29T22:19:21.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bcXQkYJDk1W2Asza5CyafSl8uxU=', 'Content-Length': '0'} 2023-09-29T22:19:21.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f76bbe32a77ce3ab-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:21.834 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.834 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.834 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.834 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.834 DEBUG:boto:Method: GET 2023-09-29T22:19:21.835 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.835 DEBUG:boto:Data: 2023-09-29T22:19:21.835 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.835 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.835 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.835 DEBUG:boto:Params: {} 2023-09-29T22:19:21.835 DEBUG:boto:Token: None 2023-09-29T22:19:21.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019b0afe3d714dfe8-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.838 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.838 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.838 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.838 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.838 DEBUG:boto:Method: GET 2023-09-29T22:19:21.839 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.839 DEBUG:boto:Data: 2023-09-29T22:19:21.839 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.839 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.839 DEBUG:boto:Params: {} 2023-09-29T22:19:21.839 DEBUG:boto:Token: None 2023-09-29T22:19:21.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b4afb8422618d63-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.842 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.842 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.842 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.842 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.842 DEBUG:boto:Method: GET 2023-09-29T22:19:21.842 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.842 DEBUG:boto:Data: 2023-09-29T22:19:21.843 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.843 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.843 DEBUG:boto:Params: {} 2023-09-29T22:19:21.843 DEBUG:boto:Token: None 2023-09-29T22:19:21.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.843 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k= 2023-09-29T22:19:21.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k=', 'Content-Length': '0'} 2023-09-29T22:19:21.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f951f82503f0d9-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.846 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.846 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.846 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.846 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.846 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.846 DEBUG:boto:Method: GET 2023-09-29T22:19:21.847 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.847 DEBUG:boto:Data: 2023-09-29T22:19:21.847 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.847 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.847 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.847 DEBUG:boto:Params: {} 2023-09-29T22:19:21.847 DEBUG:boto:Token: None 2023-09-29T22:19:21.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k= 2023-09-29T22:19:21.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k=', 'Content-Length': '0'} 2023-09-29T22:19:21.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72342629b2cf4b9-0065174d69-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.850 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.850 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.850 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.850 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.850 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:21.850 DEBUG:boto:Data: 2023-09-29T22:19:21.851 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.851 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.851 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.851 DEBUG:boto:Params: {} 2023-09-29T22:19:21.851 DEBUG:boto:Token: None 2023-09-29T22:19:21.851 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy 2023-09-29T22:19:21.851 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J8wPpbv3r8HTPDHnQpgLHRQH/cY= 2023-09-29T22:19:21.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J8wPpbv3r8HTPDHnQpgLHRQH/cY=', 'Content-Length': '0'} 2023-09-29T22:19:21.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b6a9d469b4e404d-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.853 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.853 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.854 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.854 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:21.854 DEBUG:boto:Data: 2023-09-29T22:19:21.854 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.854 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.854 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.854 DEBUG:boto:Params: {} 2023-09-29T22:19:21.854 DEBUG:boto:Token: None 2023-09-29T22:19:21.854 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy 2023-09-29T22:19:21.855 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J8wPpbv3r8HTPDHnQpgLHRQH/cY= 2023-09-29T22:19:21.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J8wPpbv3r8HTPDHnQpgLHRQH/cY=', 'Content-Length': '0'} 2023-09-29T22:19:21.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae90a8c27c2f0ff8-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:21.857 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.857 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.857 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.857 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.857 DEBUG:boto:Method: GET 2023-09-29T22:19:21.857 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.857 DEBUG:boto:Data: 2023-09-29T22:19:21.858 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.858 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.858 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.858 DEBUG:boto:Params: {} 2023-09-29T22:19:21.858 DEBUG:boto:Token: None 2023-09-29T22:19:21.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.858 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c5cb161a192d842-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.860 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:21.860 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:21.860 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.860 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.861 DEBUG:boto:Method: GET 2023-09-29T22:19:21.861 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.861 DEBUG:boto:Data: 2023-09-29T22:19:21.861 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.861 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.861 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.861 DEBUG:boto:Params: {} 2023-09-29T22:19:21.861 DEBUG:boto:Token: None 2023-09-29T22:19:21.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:21.862 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI= 2023-09-29T22:19:21.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oQC4pHPyiIzo4uhT/EtXl+jgJZI=', 'Content-Length': '0'} 2023-09-29T22:19:21.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000894f9381697e4052-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.863 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.864 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.864 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.864 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.864 DEBUG:boto:Method: GET 2023-09-29T22:19:21.864 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.864 DEBUG:boto:Data: 2023-09-29T22:19:21.864 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.864 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.864 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.865 DEBUG:boto:Params: {} 2023-09-29T22:19:21.865 DEBUG:boto:Token: None 2023-09-29T22:19:21.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.865 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k= 2023-09-29T22:19:21.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k=', 'Content-Length': '0'} 2023-09-29T22:19:21.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006f1b922b39c743e-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.867 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.867 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:21.868 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:21.868 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.868 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.868 DEBUG:boto:Method: GET 2023-09-29T22:19:21.868 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:21.868 DEBUG:boto:Data: 2023-09-29T22:19:21.868 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.868 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.868 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.869 DEBUG:boto:Params: {} 2023-09-29T22:19:21.869 DEBUG:boto:Token: None 2023-09-29T22:19:21.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/ 2023-09-29T22:19:21.869 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k= 2023-09-29T22:19:21.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OsjFcFUxpl8X53qwWGRuFvXRC8k=', 'Content-Length': '0'} 2023-09-29T22:19:21.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c5d6afa60ee998-0065174d69-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.871 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:21.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:21.872 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.872 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.872 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.872 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.872 DEBUG:boto:Method: GET 2023-09-29T22:19:21.872 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.872 DEBUG:boto:Data: 2023-09-29T22:19:21.872 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.872 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.873 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.873 DEBUG:boto:Params: {} 2023-09-29T22:19:21.873 DEBUG:boto:Token: None 2023-09-29T22:19:21.873 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.873 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs= 2023-09-29T22:19:21.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs=', 'Content-Length': '0'} 2023-09-29T22:19:21.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003238479c50ea7f8c-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.918 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.919 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.919 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.919 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.919 DEBUG:boto:Method: GET 2023-09-29T22:19:21.919 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.919 DEBUG:boto:Data: 2023-09-29T22:19:21.920 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.920 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.920 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.920 DEBUG:boto:Params: {} 2023-09-29T22:19:21.920 DEBUG:boto:Token: None 2023-09-29T22:19:21.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs= 2023-09-29T22:19:21.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs=', 'Content-Length': '0'} 2023-09-29T22:19:21.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000433202ec83f52962-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.966 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.967 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.967 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.967 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.967 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.967 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.967 DEBUG:boto:Data: 2023-09-29T22:19:21.967 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.967 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.967 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.967 DEBUG:boto:Params: {} 2023-09-29T22:19:21.968 DEBUG:boto:Token: None 2023-09-29T22:19:21.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E5xWEMcBAKWdfPzXxeIQDIKZQb8= 2023-09-29T22:19:21.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E5xWEMcBAKWdfPzXxeIQDIKZQb8=', 'Content-Length': '0'} 2023-09-29T22:19:21.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d25f2af68281f707-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.970 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.970 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.970 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.970 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.971 DEBUG:boto:Method: HEAD 2023-09-29T22:19:21.971 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.971 DEBUG:boto:Data: 2023-09-29T22:19:21.971 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:21.971 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.971 DEBUG:boto:Params: {} 2023-09-29T22:19:21.971 DEBUG:boto:Token: None 2023-09-29T22:19:21.971 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E5xWEMcBAKWdfPzXxeIQDIKZQb8= 2023-09-29T22:19:21.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E5xWEMcBAKWdfPzXxeIQDIKZQb8=', 'Content-Length': '0'} 2023-09-29T22:19:21.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000149a64c0f6338f0-0065174d69-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:21.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:21.973 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.973 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:21.974 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.974 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.974 DEBUG:boto:Method: GET 2023-09-29T22:19:21.974 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.974 DEBUG:boto:Data: 2023-09-29T22:19:21.974 DEBUG:boto:Headers: {} 2023-09-29T22:19:21.974 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:21.974 DEBUG:boto:Port: 8000 2023-09-29T22:19:21.974 DEBUG:boto:Params: {} 2023-09-29T22:19:21.975 DEBUG:boto:Token: None 2023-09-29T22:19:21.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:21 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:21.975 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs= 2023-09-29T22:19:21.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:21 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4g/Xt0T0BmKG7lhqxwbjkCC02Fs=', 'Content-Length': '0'} 2023-09-29T22:19:21.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003023068f567c08a7-0065174d69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.018 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.019 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.019 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.019 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.019 DEBUG:boto:Method: GET 2023-09-29T22:19:22.019 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.019 DEBUG:boto:Data: 2023-09-29T22:19:22.020 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.020 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.020 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.020 DEBUG:boto:Params: {} 2023-09-29T22:19:22.020 DEBUG:boto:Token: None 2023-09-29T22:19:22.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031d13b6077de41ca-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.023 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.023 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.023 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.024 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.024 DEBUG:boto:Method: GET 2023-09-29T22:19:22.024 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.024 DEBUG:boto:Data: 2023-09-29T22:19:22.024 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.024 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.024 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.024 DEBUG:boto:Params: {} 2023-09-29T22:19:22.025 DEBUG:boto:Token: None 2023-09-29T22:19:22.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003014f3bd1ec8e03f-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.027 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.028 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.028 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.028 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.028 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.028 DEBUG:boto:Method: GET 2023-09-29T22:19:22.028 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.028 DEBUG:boto:Data: 2023-09-29T22:19:22.028 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.029 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.029 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.029 DEBUG:boto:Params: {} 2023-09-29T22:19:22.029 DEBUG:boto:Token: None 2023-09-29T22:19:22.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584f3ade1207ff59-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.032 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.032 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.032 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.032 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.032 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.032 DEBUG:boto:Data: 2023-09-29T22:19:22.033 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.033 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.033 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.033 DEBUG:boto:Params: {} 2023-09-29T22:19:22.033 DEBUG:boto:Token: None 2023-09-29T22:19:22.033 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.033 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ= 2023-09-29T22:19:22.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ=', 'Content-Length': '0'} 2023-09-29T22:19:22.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e1448e3532a3f3c-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.035 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.035 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.036 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.036 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.036 DEBUG:boto:Data: 2023-09-29T22:19:22.036 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.036 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.036 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.036 DEBUG:boto:Params: {} 2023-09-29T22:19:22.036 DEBUG:boto:Token: None 2023-09-29T22:19:22.037 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.037 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ= 2023-09-29T22:19:22.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ=', 'Content-Length': '0'} 2023-09-29T22:19:22.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa0371d30841b293-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:22.039 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.039 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.039 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.039 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.039 DEBUG:boto:Method: GET 2023-09-29T22:19:22.039 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.040 DEBUG:boto:Data: 2023-09-29T22:19:22.040 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.040 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.040 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.040 DEBUG:boto:Params: {} 2023-09-29T22:19:22.040 DEBUG:boto:Token: None 2023-09-29T22:19:22.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcfdc27bfbe27ae3-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.086 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.087 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.087 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.087 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.087 DEBUG:boto:Method: GET 2023-09-29T22:19:22.087 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.087 DEBUG:boto:Data: 2023-09-29T22:19:22.087 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.087 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.087 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.088 DEBUG:boto:Params: {} 2023-09-29T22:19:22.088 DEBUG:boto:Token: None 2023-09-29T22:19:22.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4ed7bfb3319fdea-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.090 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.090 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.091 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.091 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.091 DEBUG:boto:Method: GET 2023-09-29T22:19:22.091 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.091 DEBUG:boto:Data: 2023-09-29T22:19:22.091 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.091 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.091 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.092 DEBUG:boto:Params: {} 2023-09-29T22:19:22.092 DEBUG:boto:Token: None 2023-09-29T22:19:22.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.092 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019aedccfe78aa01e-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.094 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.095 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.095 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.095 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.095 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.095 DEBUG:boto:Method: GET 2023-09-29T22:19:22.095 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.095 DEBUG:boto:Data: 2023-09-29T22:19:22.095 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.096 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.096 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.096 DEBUG:boto:Params: {} 2023-09-29T22:19:22.096 DEBUG:boto:Token: None 2023-09-29T22:19:22.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007598cfbab07a7dec-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.098 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:22.099 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.099 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.099 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.099 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.100 DEBUG:boto:Method: GET 2023-09-29T22:19:22.100 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.100 DEBUG:boto:Data: 2023-09-29T22:19:22.100 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.100 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.100 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.100 DEBUG:boto:Params: {} 2023-09-29T22:19:22.100 DEBUG:boto:Token: None 2023-09-29T22:19:22.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.101 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000596dbe6b822b6fe6-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.103 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.103 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.103 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.103 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.103 DEBUG:boto:Method: GET 2023-09-29T22:19:22.103 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.103 DEBUG:boto:Data: 2023-09-29T22:19:22.103 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.103 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.103 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.104 DEBUG:boto:Params: {} 2023-09-29T22:19:22.104 DEBUG:boto:Token: None 2023-09-29T22:19:22.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004758009427bd368e-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.147 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.331 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.331 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.333 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.333 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.333 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.333 DEBUG:boto:Data: 2023-09-29T22:19:22.333 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.334 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.334 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.334 DEBUG:boto:Params: {} 2023-09-29T22:19:22.334 DEBUG:boto:Token: None 2023-09-29T22:19:22.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.335 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vZges77P5UdmQ4DJx/XL35bbKPk= 2023-09-29T22:19:22.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vZges77P5UdmQ4DJx/XL35bbKPk=', 'Content-Length': '0'} 2023-09-29T22:19:22.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001421ea432cb0a58-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.337 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.337 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.337 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.337 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.338 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.338 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.338 DEBUG:boto:Data: 2023-09-29T22:19:22.338 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.338 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.338 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.339 DEBUG:boto:Params: {} 2023-09-29T22:19:22.339 DEBUG:boto:Token: None 2023-09-29T22:19:22.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.339 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vZges77P5UdmQ4DJx/XL35bbKPk= 2023-09-29T22:19:22.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vZges77P5UdmQ4DJx/XL35bbKPk=', 'Content-Length': '0'} 2023-09-29T22:19:22.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000989b85b8b264ba8e-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:22.342 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.342 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.343 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.343 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.343 DEBUG:boto:Method: GET 2023-09-29T22:19:22.343 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.343 DEBUG:boto:Data: 2023-09-29T22:19:22.343 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.343 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.344 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.344 DEBUG:boto:Params: {} 2023-09-29T22:19:22.344 DEBUG:boto:Token: None 2023-09-29T22:19:22.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005370c76cd4a7c8b1-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.347 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.347 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.347 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.347 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.347 DEBUG:boto:Method: GET 2023-09-29T22:19:22.348 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.348 DEBUG:boto:Data: 2023-09-29T22:19:22.348 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.348 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.348 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.348 DEBUG:boto:Params: {} 2023-09-29T22:19:22.349 DEBUG:boto:Token: None 2023-09-29T22:19:22.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.349 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfc4c1173db1fb40-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.394 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.395 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.395 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.395 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.395 DEBUG:boto:Method: GET 2023-09-29T22:19:22.395 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.395 DEBUG:boto:Data: 2023-09-29T22:19:22.395 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.395 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.395 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.395 DEBUG:boto:Params: {} 2023-09-29T22:19:22.396 DEBUG:boto:Token: None 2023-09-29T22:19:22.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9d5b0f571903c5-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.398 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.399 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.399 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.399 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.399 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.399 DEBUG:boto:Method: GET 2023-09-29T22:19:22.399 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.399 DEBUG:boto:Data: 2023-09-29T22:19:22.399 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.400 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.400 DEBUG:boto:Params: {} 2023-09-29T22:19:22.400 DEBUG:boto:Token: None 2023-09-29T22:19:22.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804474891b6e19b6-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.402 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.403 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.403 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.403 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.403 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.403 DEBUG:boto:Data: 2023-09-29T22:19:22.403 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.403 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.404 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.404 DEBUG:boto:Params: {} 2023-09-29T22:19:22.404 DEBUG:boto:Token: None 2023-09-29T22:19:22.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nfErO0wkvX+JqOnqRmycXgUy2z8= 2023-09-29T22:19:22.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nfErO0wkvX+JqOnqRmycXgUy2z8=', 'Content-Length': '0'} 2023-09-29T22:19:22.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc45d29bc6f7d577-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.406 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.406 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.406 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.407 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.407 DEBUG:boto:Data: 2023-09-29T22:19:22.407 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.407 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.407 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.407 DEBUG:boto:Params: {} 2023-09-29T22:19:22.407 DEBUG:boto:Token: None 2023-09-29T22:19:22.407 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nfErO0wkvX+JqOnqRmycXgUy2z8= 2023-09-29T22:19:22.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nfErO0wkvX+JqOnqRmycXgUy2z8=', 'Content-Length': '0'} 2023-09-29T22:19:22.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1f48694f19a9e9-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:22.410 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.410 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.410 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.411 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.411 DEBUG:boto:Method: GET 2023-09-29T22:19:22.411 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.411 DEBUG:boto:Data: 2023-09-29T22:19:22.411 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.412 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.412 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.412 DEBUG:boto:Params: {} 2023-09-29T22:19:22.412 DEBUG:boto:Token: None 2023-09-29T22:19:22.413 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.413 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e1d2518de69473c-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.415 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.415 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.415 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.415 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.415 DEBUG:boto:Method: GET 2023-09-29T22:19:22.415 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.416 DEBUG:boto:Data: 2023-09-29T22:19:22.416 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.416 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.416 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.416 DEBUG:boto:Params: {} 2023-09-29T22:19:22.416 DEBUG:boto:Token: None 2023-09-29T22:19:22.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2f4d6fc24cc714e-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.458 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.459 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.459 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.459 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.459 DEBUG:boto:Method: GET 2023-09-29T22:19:22.459 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.459 DEBUG:boto:Data: 2023-09-29T22:19:22.459 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.459 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.460 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.460 DEBUG:boto:Params: {} 2023-09-29T22:19:22.460 DEBUG:boto:Token: None 2023-09-29T22:19:22.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d5062d5fbd2705a-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.462 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.463 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.463 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.463 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.463 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.464 DEBUG:boto:Method: GET 2023-09-29T22:19:22.464 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:22.464 DEBUG:boto:Data: 2023-09-29T22:19:22.464 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.464 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.464 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.464 DEBUG:boto:Params: {} 2023-09-29T22:19:22.464 DEBUG:boto:Token: None 2023-09-29T22:19:22.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.465 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935bb2a004ebe4f1-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.467 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:22.468 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.468 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.468 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.468 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.468 DEBUG:boto:Method: GET 2023-09-29T22:19:22.468 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.468 DEBUG:boto:Data: 2023-09-29T22:19:22.468 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.469 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.469 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.469 DEBUG:boto:Params: {} 2023-09-29T22:19:22.469 DEBUG:boto:Token: None 2023-09-29T22:19:22.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d18ee81b3418503e-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.471 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.471 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.471 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.472 DEBUG:boto:Method: GET 2023-09-29T22:19:22.472 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.472 DEBUG:boto:Data: 2023-09-29T22:19:22.472 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.472 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.472 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.472 DEBUG:boto:Params: {} 2023-09-29T22:19:22.472 DEBUG:boto:Token: None 2023-09-29T22:19:22.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf27cb473e9cf9c-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.474 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.475 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.475 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.475 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.475 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.475 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.475 DEBUG:boto:Data: 2023-09-29T22:19:22.475 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.475 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.475 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.476 DEBUG:boto:Params: {} 2023-09-29T22:19:22.476 DEBUG:boto:Token: None 2023-09-29T22:19:22.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5w8D/PZGEncvYK4g2/kbYoVC1nA= 2023-09-29T22:19:22.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5w8D/PZGEncvYK4g2/kbYoVC1nA=', 'Content-Length': '0'} 2023-09-29T22:19:22.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee2d54358b357e68-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.478 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.478 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.478 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.478 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.478 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.478 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.479 DEBUG:boto:Data: 2023-09-29T22:19:22.479 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.479 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.479 DEBUG:boto:Params: {} 2023-09-29T22:19:22.479 DEBUG:boto:Token: None 2023-09-29T22:19:22.479 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.479 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5w8D/PZGEncvYK4g2/kbYoVC1nA= 2023-09-29T22:19:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5w8D/PZGEncvYK4g2/kbYoVC1nA=', 'Content-Length': '0'} 2023-09-29T22:19:22.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b065d7f19af3c26f-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:22.481 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.481 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.481 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.482 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.482 DEBUG:boto:Method: GET 2023-09-29T22:19:22.482 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.482 DEBUG:boto:Data: 2023-09-29T22:19:22.482 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.482 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.482 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.482 DEBUG:boto:Params: {} 2023-09-29T22:19:22.482 DEBUG:boto:Token: None 2023-09-29T22:19:22.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000261cd78e46a4a36f-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.530 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.531 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.531 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.531 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.531 DEBUG:boto:Method: GET 2023-09-29T22:19:22.531 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.531 DEBUG:boto:Data: 2023-09-29T22:19:22.531 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.531 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.531 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.531 DEBUG:boto:Params: {} 2023-09-29T22:19:22.532 DEBUG:boto:Token: None 2023-09-29T22:19:22.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000458840e9d29b8de9-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.575 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.575 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.575 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.576 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.576 DEBUG:boto:Method: GET 2023-09-29T22:19:22.576 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.576 DEBUG:boto:Data: 2023-09-29T22:19:22.577 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.577 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.577 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.578 DEBUG:boto:Params: {} 2023-09-29T22:19:22.578 DEBUG:boto:Token: None 2023-09-29T22:19:22.578 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c4126f4c1c562ec-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.582 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.582 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.583 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.583 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.583 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.584 DEBUG:boto:Method: GET 2023-09-29T22:19:22.584 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.584 DEBUG:boto:Data: 2023-09-29T22:19:22.584 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.584 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.585 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.585 DEBUG:boto:Params: {} 2023-09-29T22:19:22.585 DEBUG:boto:Token: None 2023-09-29T22:19:22.586 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.586 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc2add5a53ac19dc-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.589 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.589 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.590 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.590 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.590 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.590 DEBUG:boto:Data: 2023-09-29T22:19:22.591 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.591 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.591 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.591 DEBUG:boto:Params: {} 2023-09-29T22:19:22.592 DEBUG:boto:Token: None 2023-09-29T22:19:22.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6TsF9Ndf+t+pJbInxjHzr1EXuHo= 2023-09-29T22:19:22.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6TsF9Ndf+t+pJbInxjHzr1EXuHo=', 'Content-Length': '0'} 2023-09-29T22:19:22.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000861b089a9939e142-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.595 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.595 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.596 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.596 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.596 DEBUG:boto:Data: 2023-09-29T22:19:22.596 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.596 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.597 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.597 DEBUG:boto:Params: {} 2023-09-29T22:19:22.597 DEBUG:boto:Token: None 2023-09-29T22:19:22.597 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6TsF9Ndf+t+pJbInxjHzr1EXuHo= 2023-09-29T22:19:22.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6TsF9Ndf+t+pJbInxjHzr1EXuHo=', 'Content-Length': '0'} 2023-09-29T22:19:22.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc306287e86b85a8-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:22.601 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.601 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.602 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.602 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.602 DEBUG:boto:Method: GET 2023-09-29T22:19:22.602 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.602 DEBUG:boto:Data: 2023-09-29T22:19:22.603 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.603 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.603 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.603 DEBUG:boto:Params: {} 2023-09-29T22:19:22.604 DEBUG:boto:Token: None 2023-09-29T22:19:22.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e579b10f9addcde-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.650 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.651 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:22.651 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.651 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.652 DEBUG:boto:Method: GET 2023-09-29T22:19:22.652 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.652 DEBUG:boto:Data: 2023-09-29T22:19:22.652 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.652 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.653 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.653 DEBUG:boto:Params: {} 2023-09-29T22:19:22.653 DEBUG:boto:Token: None 2023-09-29T22:19:22.654 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:22.654 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0= 2023-09-29T22:19:22.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3k/RrhxzkhZLJ41pdkK/l41Gv0=', 'Content-Length': '0'} 2023-09-29T22:19:22.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000628c840b64317981-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.698 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.699 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.699 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.699 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.699 DEBUG:boto:Method: GET 2023-09-29T22:19:22.699 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.699 DEBUG:boto:Data: 2023-09-29T22:19:22.699 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.699 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:22.699 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.700 DEBUG:boto:Params: {} 2023-09-29T22:19:22.700 DEBUG:boto:Token: None 2023-09-29T22:19:22.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6ce0b80277591e1-0065174d6a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.703 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.703 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.703 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.703 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.704 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.704 DEBUG:boto:Method: GET 2023-09-29T22:19:22.704 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:22.704 DEBUG:boto:Data: 2023-09-29T22:19:22.704 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.704 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.704 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.705 DEBUG:boto:Params: {} 2023-09-29T22:19:22.705 DEBUG:boto:Token: None 2023-09-29T22:19:22.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.705 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d81a1f03d57f578-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.707 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone2, test-zone1} 2023-09-29T22:19:22.708 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone3, test-zone1} 2023-09-29T22:19:22.708 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.708 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.708 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.708 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:22.708 DEBUG:boto:Data: 2023-09-29T22:19:22.708 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.709 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.709 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.709 DEBUG:boto:Params: {} 2023-09-29T22:19:22.709 DEBUG:boto:Token: None 2023-09-29T22:19:22.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z8O4rZNGJcSbJi0j4Apghdm4AnY= 2023-09-29T22:19:22.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z8O4rZNGJcSbJi0j4Apghdm4AnY=', 'Content-Length': '0'} 2023-09-29T22:19:22.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033bb456648a63b7d-0065174d6a-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.711 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.711 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.711 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.711 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:22.712 DEBUG:boto:Data: 2023-09-29T22:19:22.712 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.712 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.712 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.712 DEBUG:boto:Params: {} 2023-09-29T22:19:22.712 DEBUG:boto:Token: None 2023-09-29T22:19:22.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:z8O4rZNGJcSbJi0j4Apghdm4AnY= 2023-09-29T22:19:22.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:z8O4rZNGJcSbJi0j4Apghdm4AnY=', 'Content-Length': '0'} 2023-09-29T22:19:22.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008025f695f82a16b3-0065174d6a-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.714 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:22.715 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.715 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.715 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.715 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.715 DEBUG:boto:Method: GET 2023-09-29T22:19:22.715 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.715 DEBUG:boto:Data: 2023-09-29T22:19:22.715 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.715 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.716 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.716 DEBUG:boto:Params: {} 2023-09-29T22:19:22.716 DEBUG:boto:Token: None 2023-09-29T22:19:22.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/?versions 2023-09-29T22:19:22.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY= 2023-09-29T22:19:22.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY=', 'Content-Length': '0'} 2023-09-29T22:19:22.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f303c3ee2c5d3833-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.718 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.719 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:22.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:22.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:22.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:22.719 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:22.719 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.719 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.720 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.720 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.720 DEBUG:boto:Method: GET 2023-09-29T22:19:22.720 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.720 DEBUG:boto:Data: 2023-09-29T22:19:22.720 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.720 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.720 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.720 DEBUG:boto:Params: {} 2023-09-29T22:19:22.721 DEBUG:boto:Token: None 2023-09-29T22:19:22.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/?versions 2023-09-29T22:19:22.721 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY= 2023-09-29T22:19:22.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY=', 'Content-Length': '0'} 2023-09-29T22:19:22.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce05741757fef91b-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.723 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.724 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:22.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:22.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:22.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:22.724 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.725 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.725 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.725 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.725 DEBUG:boto:Method: GET 2023-09-29T22:19:22.725 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.725 DEBUG:boto:Data: 2023-09-29T22:19:22.725 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.725 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.725 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.725 DEBUG:boto:Params: {} 2023-09-29T22:19:22.726 DEBUG:boto:Token: None 2023-09-29T22:19:22.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/?versions 2023-09-29T22:19:22.726 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY= 2023-09-29T22:19:22.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY=', 'Content-Length': '0'} 2023-09-29T22:19:22.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c347908211fa44-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.728 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.729 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.729 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.729 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.729 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.729 DEBUG:boto:Method: GET 2023-09-29T22:19:22.729 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:22.730 DEBUG:boto:Data: 2023-09-29T22:19:22.730 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.730 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.730 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.730 DEBUG:boto:Params: {} 2023-09-29T22:19:22.730 DEBUG:boto:Token: None 2023-09-29T22:19:22.730 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/?versions 2023-09-29T22:19:22.730 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY= 2023-09-29T22:19:22.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwqoScJAzGFy+HSCZm2jloFLDNY=', 'Content-Length': '0'} 2023-09-29T22:19:22.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e47fd00f747fc2e-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.733 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:22.734 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.734 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.735 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.735 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.735 DEBUG:boto:Method: GET 2023-09-29T22:19:22.735 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.736 DEBUG:boto:Data: 2023-09-29T22:19:22.736 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.736 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.736 DEBUG:boto:Params: {} 2023-09-29T22:19:22.736 DEBUG:boto:Token: None 2023-09-29T22:19:22.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f09214a04acce76-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.739 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.739 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.739 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.739 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.739 DEBUG:boto:Method: GET 2023-09-29T22:19:22.739 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.739 DEBUG:boto:Data: 2023-09-29T22:19:22.739 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.741 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.741 DEBUG:boto:Params: {} 2023-09-29T22:19:22.741 DEBUG:boto:Token: None 2023-09-29T22:19:22.741 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dc9960ec4107422-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.743 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.744 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.744 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.744 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.744 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.744 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.744 DEBUG:boto:Data: 2023-09-29T22:19:22.744 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.744 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.744 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.744 DEBUG:boto:Params: {} 2023-09-29T22:19:22.745 DEBUG:boto:Token: None 2023-09-29T22:19:22.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3gg5j+4thI8IQ9Yv+vBzulvDcU8= 2023-09-29T22:19:22.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3gg5j+4thI8IQ9Yv+vBzulvDcU8=', 'Content-Length': '0'} 2023-09-29T22:19:22.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2da82c4e4181520-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.746 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.747 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.747 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.747 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.747 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.747 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.747 DEBUG:boto:Data: 2023-09-29T22:19:22.747 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.747 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.748 DEBUG:boto:Params: {} 2023-09-29T22:19:22.748 DEBUG:boto:Token: None 2023-09-29T22:19:22.748 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3gg5j+4thI8IQ9Yv+vBzulvDcU8= 2023-09-29T22:19:22.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3gg5j+4thI8IQ9Yv+vBzulvDcU8=', 'Content-Length': '0'} 2023-09-29T22:19:22.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c3b944c261e6f69-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:22.750 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.750 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.750 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.750 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.750 DEBUG:boto:Method: GET 2023-09-29T22:19:22.750 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.751 DEBUG:boto:Data: 2023-09-29T22:19:22.751 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.751 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.751 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.751 DEBUG:boto:Params: {} 2023-09-29T22:19:22.751 DEBUG:boto:Token: None 2023-09-29T22:19:22.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3b571502b726cd4-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.753 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.753 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.753 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.753 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.754 DEBUG:boto:Method: GET 2023-09-29T22:19:22.754 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.754 DEBUG:boto:Data: 2023-09-29T22:19:22.754 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.754 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.754 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.754 DEBUG:boto:Params: {} 2023-09-29T22:19:22.754 DEBUG:boto:Token: None 2023-09-29T22:19:22.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.755 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee0e96b7475afc37-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.756 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.756 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.757 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.757 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.757 DEBUG:boto:Method: GET 2023-09-29T22:19:22.757 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.757 DEBUG:boto:Data: 2023-09-29T22:19:22.757 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.757 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.757 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.757 DEBUG:boto:Params: {} 2023-09-29T22:19:22.757 DEBUG:boto:Token: None 2023-09-29T22:19:22.758 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.758 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d9c9b8832f79f5-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.802 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.803 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.803 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.803 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.803 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.803 DEBUG:boto:Method: GET 2023-09-29T22:19:22.803 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.803 DEBUG:boto:Data: 2023-09-29T22:19:22.804 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.804 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.804 DEBUG:boto:Params: {} 2023-09-29T22:19:22.804 DEBUG:boto:Token: None 2023-09-29T22:19:22.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.804 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f6ece6e29ac5476-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.807 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.807 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.807 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.807 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.807 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:22.807 DEBUG:boto:Data: 2023-09-29T22:19:22.807 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.808 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.808 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.808 DEBUG:boto:Params: {} 2023-09-29T22:19:22.808 DEBUG:boto:Token: None 2023-09-29T22:19:22.808 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy 2023-09-29T22:19:22.808 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bCXGw1PvzSqWrwq9RUGJvOJVY4s= 2023-09-29T22:19:22.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bCXGw1PvzSqWrwq9RUGJvOJVY4s=', 'Content-Length': '0'} 2023-09-29T22:19:22.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007258b1f070b33e8d-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.810 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.810 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.810 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.811 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:22.811 DEBUG:boto:Data: 2023-09-29T22:19:22.811 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.811 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.811 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.811 DEBUG:boto:Params: {} 2023-09-29T22:19:22.811 DEBUG:boto:Token: None 2023-09-29T22:19:22.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy 2023-09-29T22:19:22.811 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bCXGw1PvzSqWrwq9RUGJvOJVY4s= 2023-09-29T22:19:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bCXGw1PvzSqWrwq9RUGJvOJVY4s=', 'Content-Length': '0'} 2023-09-29T22:19:22.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004094d379533d54d0-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:22.814 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.814 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.814 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.814 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.814 DEBUG:boto:Method: GET 2023-09-29T22:19:22.814 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.814 DEBUG:boto:Data: 2023-09-29T22:19:22.814 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.814 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.814 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.815 DEBUG:boto:Params: {} 2023-09-29T22:19:22.815 DEBUG:boto:Token: None 2023-09-29T22:19:22.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.815 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095c6b2167893632d-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.817 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:22.817 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:22.817 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.817 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.817 DEBUG:boto:Method: GET 2023-09-29T22:19:22.817 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.817 DEBUG:boto:Data: 2023-09-29T22:19:22.817 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.817 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.817 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.818 DEBUG:boto:Params: {} 2023-09-29T22:19:22.818 DEBUG:boto:Token: None 2023-09-29T22:19:22.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:22.818 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg= 2023-09-29T22:19:22.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jvtc3dq28Bfpw6pbDhp6UAyt2cg=', 'Content-Length': '0'} 2023-09-29T22:19:22.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa1cf0837b738c31-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.820 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.820 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.820 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.820 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.820 DEBUG:boto:Method: GET 2023-09-29T22:19:22.820 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.820 DEBUG:boto:Data: 2023-09-29T22:19:22.821 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.821 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.821 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.821 DEBUG:boto:Params: {} 2023-09-29T22:19:22.821 DEBUG:boto:Token: None 2023-09-29T22:19:22.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.821 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d88435003d72e21-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.824 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.824 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.824 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.824 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.824 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.824 DEBUG:boto:Method: GET 2023-09-29T22:19:22.824 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:22.825 DEBUG:boto:Data: 2023-09-29T22:19:22.825 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.825 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.825 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.825 DEBUG:boto:Params: {} 2023-09-29T22:19:22.825 DEBUG:boto:Token: None 2023-09-29T22:19:22.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.825 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bc7c223c04f533-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.828 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:22.828 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.828 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.828 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.828 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.828 DEBUG:boto:Method: GET 2023-09-29T22:19:22.829 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.829 DEBUG:boto:Data: 2023-09-29T22:19:22.829 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.829 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.829 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.829 DEBUG:boto:Params: {} 2023-09-29T22:19:22.829 DEBUG:boto:Token: None 2023-09-29T22:19:22.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001103f87cf34e1e3c-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.831 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.831 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.831 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.831 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.832 DEBUG:boto:Method: GET 2023-09-29T22:19:22.832 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.832 DEBUG:boto:Data: 2023-09-29T22:19:22.832 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.832 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.832 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.832 DEBUG:boto:Params: {} 2023-09-29T22:19:22.832 DEBUG:boto:Token: None 2023-09-29T22:19:22.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.833 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036b960112dbf74fd-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.835 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.835 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.835 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.835 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.835 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.835 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.835 DEBUG:boto:Data: 2023-09-29T22:19:22.835 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.835 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.835 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.836 DEBUG:boto:Params: {} 2023-09-29T22:19:22.836 DEBUG:boto:Token: None 2023-09-29T22:19:22.836 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8TSYcsvX6MCp6ATrlmKCHwJi4hk= 2023-09-29T22:19:22.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8TSYcsvX6MCp6ATrlmKCHwJi4hk=', 'Content-Length': '0'} 2023-09-29T22:19:22.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea1dbab063240acc-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.838 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.838 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.838 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.838 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.838 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.838 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.838 DEBUG:boto:Data: 2023-09-29T22:19:22.838 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.839 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.839 DEBUG:boto:Params: {} 2023-09-29T22:19:22.839 DEBUG:boto:Token: None 2023-09-29T22:19:22.839 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.839 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8TSYcsvX6MCp6ATrlmKCHwJi4hk= 2023-09-29T22:19:22.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8TSYcsvX6MCp6ATrlmKCHwJi4hk=', 'Content-Length': '0'} 2023-09-29T22:19:22.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033ee7436886633d1-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:22.841 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.841 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.841 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.841 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.841 DEBUG:boto:Method: GET 2023-09-29T22:19:22.842 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.842 DEBUG:boto:Data: 2023-09-29T22:19:22.842 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.842 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.842 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.842 DEBUG:boto:Params: {} 2023-09-29T22:19:22.842 DEBUG:boto:Token: None 2023-09-29T22:19:22.842 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a411089ba456c4ed-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.886 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.887 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.887 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.887 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.887 DEBUG:boto:Method: GET 2023-09-29T22:19:22.887 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.887 DEBUG:boto:Data: 2023-09-29T22:19:22.887 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.887 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.887 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.888 DEBUG:boto:Params: {} 2023-09-29T22:19:22.888 DEBUG:boto:Token: None 2023-09-29T22:19:22.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a597dc04938ec5b-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.890 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.890 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.891 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.891 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.891 DEBUG:boto:Method: GET 2023-09-29T22:19:22.891 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.891 DEBUG:boto:Data: 2023-09-29T22:19:22.891 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.891 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.891 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.891 DEBUG:boto:Params: {} 2023-09-29T22:19:22.891 DEBUG:boto:Token: None 2023-09-29T22:19:22.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4b47d9978759799-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.894 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.894 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.894 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.895 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.895 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.895 DEBUG:boto:Method: GET 2023-09-29T22:19:22.895 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.895 DEBUG:boto:Data: 2023-09-29T22:19:22.895 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.895 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.895 DEBUG:boto:Params: {} 2023-09-29T22:19:22.895 DEBUG:boto:Token: None 2023-09-29T22:19:22.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec4dd18e20d7e80-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.898 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.899 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.899 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.899 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.899 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.899 DEBUG:boto:Data: 2023-09-29T22:19:22.899 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.899 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.899 DEBUG:boto:Params: {} 2023-09-29T22:19:22.900 DEBUG:boto:Token: None 2023-09-29T22:19:22.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ= 2023-09-29T22:19:22.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ=', 'Content-Length': '0'} 2023-09-29T22:19:22.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000981efa92d3300f12-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.902 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.902 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.902 DEBUG:boto:Method: HEAD 2023-09-29T22:19:22.902 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.902 DEBUG:boto:Data: 2023-09-29T22:19:22.903 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.903 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.903 DEBUG:boto:Params: {} 2023-09-29T22:19:22.903 DEBUG:boto:Token: None 2023-09-29T22:19:22.903 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.903 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ= 2023-09-29T22:19:22.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qJvoO0A+/qIs8JBvrPK5+XfyxuQ=', 'Content-Length': '0'} 2023-09-29T22:19:22.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b74aef785438771-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:22.905 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.905 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.905 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.906 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.906 DEBUG:boto:Method: GET 2023-09-29T22:19:22.906 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.906 DEBUG:boto:Data: 2023-09-29T22:19:22.906 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.906 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.906 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.906 DEBUG:boto:Params: {} 2023-09-29T22:19:22.906 DEBUG:boto:Token: None 2023-09-29T22:19:22.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.907 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000627a8f7d2edc1e63-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.954 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.955 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:22.955 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.955 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.955 DEBUG:boto:Method: GET 2023-09-29T22:19:22.955 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.955 DEBUG:boto:Data: 2023-09-29T22:19:22.955 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.955 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.955 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.956 DEBUG:boto:Params: {} 2023-09-29T22:19:22.956 DEBUG:boto:Token: None 2023-09-29T22:19:22.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:22.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w= 2023-09-29T22:19:22.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uctbPHXiBuhczU8ASIj8PeXIY0w=', 'Content-Length': '0'} 2023-09-29T22:19:22.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000375b195b9e51ae9-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.958 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.958 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.959 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.959 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.959 DEBUG:boto:Method: GET 2023-09-29T22:19:22.959 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.959 DEBUG:boto:Data: 2023-09-29T22:19:22.959 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.959 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.959 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.959 DEBUG:boto:Params: {} 2023-09-29T22:19:22.959 DEBUG:boto:Token: None 2023-09-29T22:19:22.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098684d9f6de1767a-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.962 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.962 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:22.963 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:22.963 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.963 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.963 DEBUG:boto:Method: GET 2023-09-29T22:19:22.963 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:22.963 DEBUG:boto:Data: 2023-09-29T22:19:22.963 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.963 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.963 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.963 DEBUG:boto:Params: {} 2023-09-29T22:19:22.964 DEBUG:boto:Token: None 2023-09-29T22:19:22.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/ 2023-09-29T22:19:22.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo= 2023-09-29T22:19:22.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UOci99RLJ6LjbTAILFAGACCJfPo=', 'Content-Length': '0'} 2023-09-29T22:19:22.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee4ff487cd7492a-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.966 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:22.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:22.967 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.967 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.967 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.967 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.967 DEBUG:boto:Method: GET 2023-09-29T22:19:22.967 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.967 DEBUG:boto:Data: 2023-09-29T22:19:22.967 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.968 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:22.968 DEBUG:boto:Port: 8001 2023-09-29T22:19:22.968 DEBUG:boto:Params: {} 2023-09-29T22:19:22.968 DEBUG:boto:Token: None 2023-09-29T22:19:22.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067dbf3ef136982dc-0065174d6a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:22.970 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.970 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:22.970 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.970 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.970 DEBUG:boto:Method: GET 2023-09-29T22:19:22.970 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.971 DEBUG:boto:Data: 2023-09-29T22:19:22.971 DEBUG:boto:Headers: {} 2023-09-29T22:19:22.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:22.971 DEBUG:boto:Port: 8000 2023-09-29T22:19:22.971 DEBUG:boto:Params: {} 2023-09-29T22:19:22.971 DEBUG:boto:Token: None 2023-09-29T22:19:22.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:22 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:22.971 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE= 2023-09-29T22:19:22.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G/WisiBGBDw6OfSXSuZC/qwCjBE=', 'Content-Length': '0'} 2023-09-29T22:19:22.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3a22a3ac87fe009-0065174d6a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.014 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.015 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.015 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.015 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.015 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.015 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.015 DEBUG:boto:Data: 2023-09-29T22:19:23.015 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.015 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.015 DEBUG:boto:Params: {} 2023-09-29T22:19:23.016 DEBUG:boto:Token: None 2023-09-29T22:19:23.016 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4= 2023-09-29T22:19:23.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4=', 'Content-Length': '0'} 2023-09-29T22:19:23.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001572d4c1f6111dba-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.018 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.018 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.018 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.019 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.019 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.019 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.019 DEBUG:boto:Data: 2023-09-29T22:19:23.019 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.019 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.019 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.019 DEBUG:boto:Params: {} 2023-09-29T22:19:23.019 DEBUG:boto:Token: None 2023-09-29T22:19:23.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4= 2023-09-29T22:19:23.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4=', 'Content-Length': '0'} 2023-09-29T22:19:23.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c49f19b6402b7b4-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:23.021 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.022 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.022 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.022 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.022 DEBUG:boto:Method: GET 2023-09-29T22:19:23.022 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.022 DEBUG:boto:Data: 2023-09-29T22:19:23.022 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.022 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.022 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.023 DEBUG:boto:Params: {} 2023-09-29T22:19:23.023 DEBUG:boto:Token: None 2023-09-29T22:19:23.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.023 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022cfd1e06878f1c0-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.066 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.066 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.067 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.067 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.067 DEBUG:boto:Method: GET 2023-09-29T22:19:23.067 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.067 DEBUG:boto:Data: 2023-09-29T22:19:23.067 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.067 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.067 DEBUG:boto:Params: {} 2023-09-29T22:19:23.068 DEBUG:boto:Token: None 2023-09-29T22:19:23.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cab999f513724040-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.110 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.111 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.111 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.111 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.111 DEBUG:boto:Method: GET 2023-09-29T22:19:23.111 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.111 DEBUG:boto:Data: 2023-09-29T22:19:23.111 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.111 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.112 DEBUG:boto:Params: {} 2023-09-29T22:19:23.112 DEBUG:boto:Token: None 2023-09-29T22:19:23.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479c050c89b02b29-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.158 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.159 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.159 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.159 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.159 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.159 DEBUG:boto:Method: GET 2023-09-29T22:19:23.160 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.160 DEBUG:boto:Data: 2023-09-29T22:19:23.160 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.160 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.160 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.160 DEBUG:boto:Params: {} 2023-09-29T22:19:23.160 DEBUG:boto:Token: None 2023-09-29T22:19:23.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.160 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9757c1c3a7c47c1-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.163 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.163 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.164 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.164 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.164 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.164 DEBUG:boto:Data: 2023-09-29T22:19:23.164 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.164 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.164 DEBUG:boto:Params: {} 2023-09-29T22:19:23.164 DEBUG:boto:Token: None 2023-09-29T22:19:23.165 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SQR2hfPbsutVRyL4rgRkVY5kSqA= 2023-09-29T22:19:23.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SQR2hfPbsutVRyL4rgRkVY5kSqA=', 'Content-Length': '0'} 2023-09-29T22:19:23.167 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c74d49fa2935ad2f-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.167 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.167 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.167 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.167 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.168 DEBUG:boto:Data: 2023-09-29T22:19:23.168 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.168 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.168 DEBUG:boto:Params: {} 2023-09-29T22:19:23.168 DEBUG:boto:Token: None 2023-09-29T22:19:23.168 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SQR2hfPbsutVRyL4rgRkVY5kSqA= 2023-09-29T22:19:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SQR2hfPbsutVRyL4rgRkVY5kSqA=', 'Content-Length': '0'} 2023-09-29T22:19:23.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e22775c390cf1ed-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:23.171 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.171 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.171 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.171 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.172 DEBUG:boto:Method: GET 2023-09-29T22:19:23.172 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.172 DEBUG:boto:Data: 2023-09-29T22:19:23.172 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.172 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.172 DEBUG:boto:Params: {} 2023-09-29T22:19:23.172 DEBUG:boto:Token: None 2023-09-29T22:19:23.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000083f90b0e96ab046-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.218 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.219 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.219 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.219 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.219 DEBUG:boto:Method: GET 2023-09-29T22:19:23.219 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.219 DEBUG:boto:Data: 2023-09-29T22:19:23.219 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.219 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.219 DEBUG:boto:Params: {} 2023-09-29T22:19:23.220 DEBUG:boto:Token: None 2023-09-29T22:19:23.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004030faf81e96bc20-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.262 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.263 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.263 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.263 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.263 DEBUG:boto:Method: GET 2023-09-29T22:19:23.263 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.263 DEBUG:boto:Data: 2023-09-29T22:19:23.263 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.264 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.264 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.264 DEBUG:boto:Params: {} 2023-09-29T22:19:23.264 DEBUG:boto:Token: None 2023-09-29T22:19:23.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ea8e8010bc7105-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.314 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.315 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.315 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.315 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.315 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.315 DEBUG:boto:Method: GET 2023-09-29T22:19:23.315 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:23.315 DEBUG:boto:Data: 2023-09-29T22:19:23.316 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.316 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.316 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.316 DEBUG:boto:Params: {} 2023-09-29T22:19:23.316 DEBUG:boto:Token: None 2023-09-29T22:19:23.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b1bb18e01e3c3b-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.362 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:23.363 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.363 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.363 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.363 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.363 DEBUG:boto:Method: GET 2023-09-29T22:19:23.363 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.363 DEBUG:boto:Data: 2023-09-29T22:19:23.364 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.364 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.364 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.364 DEBUG:boto:Params: {} 2023-09-29T22:19:23.364 DEBUG:boto:Token: None 2023-09-29T22:19:23.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.364 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011b1ba3b0fac59da-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.366 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.367 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.367 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.367 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.367 DEBUG:boto:Method: GET 2023-09-29T22:19:23.367 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.367 DEBUG:boto:Data: 2023-09-29T22:19:23.367 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.367 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.367 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.367 DEBUG:boto:Params: {} 2023-09-29T22:19:23.368 DEBUG:boto:Token: None 2023-09-29T22:19:23.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b0aff4671d041dc-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.370 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.370 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.370 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.370 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.370 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.370 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.370 DEBUG:boto:Data: 2023-09-29T22:19:23.371 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.371 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.371 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.371 DEBUG:boto:Params: {} 2023-09-29T22:19:23.371 DEBUG:boto:Token: None 2023-09-29T22:19:23.371 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.371 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:scGWHJhBdqXn5kO78AhgRJMcylA= 2023-09-29T22:19:23.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:scGWHJhBdqXn5kO78AhgRJMcylA=', 'Content-Length': '0'} 2023-09-29T22:19:23.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000981ded1ac8fdc261-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.373 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.373 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.373 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.373 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.374 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.374 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.374 DEBUG:boto:Data: 2023-09-29T22:19:23.374 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.374 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.374 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.374 DEBUG:boto:Params: {} 2023-09-29T22:19:23.374 DEBUG:boto:Token: None 2023-09-29T22:19:23.374 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.375 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:scGWHJhBdqXn5kO78AhgRJMcylA= 2023-09-29T22:19:23.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:scGWHJhBdqXn5kO78AhgRJMcylA=', 'Content-Length': '0'} 2023-09-29T22:19:23.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000109f7c6e6ac6d624-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:23.377 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.377 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.377 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.377 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.377 DEBUG:boto:Method: GET 2023-09-29T22:19:23.378 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.378 DEBUG:boto:Data: 2023-09-29T22:19:23.378 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.378 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.378 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.378 DEBUG:boto:Params: {} 2023-09-29T22:19:23.378 DEBUG:boto:Token: None 2023-09-29T22:19:23.378 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.379 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8f68fc6a2db2f70-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.381 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.381 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.381 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.381 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.381 DEBUG:boto:Method: GET 2023-09-29T22:19:23.381 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.382 DEBUG:boto:Data: 2023-09-29T22:19:23.382 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.382 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.382 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.382 DEBUG:boto:Params: {} 2023-09-29T22:19:23.382 DEBUG:boto:Token: None 2023-09-29T22:19:23.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b57c64fd1dba76a-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.426 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.427 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.427 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.427 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.427 DEBUG:boto:Method: GET 2023-09-29T22:19:23.427 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.427 DEBUG:boto:Data: 2023-09-29T22:19:23.427 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.427 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.427 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.427 DEBUG:boto:Params: {} 2023-09-29T22:19:23.428 DEBUG:boto:Token: None 2023-09-29T22:19:23.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c6467601628e58-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.430 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.431 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.431 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.431 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.431 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.431 DEBUG:boto:Method: GET 2023-09-29T22:19:23.431 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.431 DEBUG:boto:Data: 2023-09-29T22:19:23.432 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.432 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.432 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.432 DEBUG:boto:Params: {} 2023-09-29T22:19:23.432 DEBUG:boto:Token: None 2023-09-29T22:19:23.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.432 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e7618b91cf4b29-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.434 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.435 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.435 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.435 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.435 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.435 DEBUG:boto:Data: 2023-09-29T22:19:23.435 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.436 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.436 DEBUG:boto:Params: {} 2023-09-29T22:19:23.436 DEBUG:boto:Token: None 2023-09-29T22:19:23.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.436 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G1Lw+bKKm77x4xbw0lI4q+DzM+8= 2023-09-29T22:19:23.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G1Lw+bKKm77x4xbw0lI4q+DzM+8=', 'Content-Length': '0'} 2023-09-29T22:19:23.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1407662193910eb-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.438 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.438 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.438 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.439 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.439 DEBUG:boto:Data: 2023-09-29T22:19:23.439 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.439 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.439 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.439 DEBUG:boto:Params: {} 2023-09-29T22:19:23.439 DEBUG:boto:Token: None 2023-09-29T22:19:23.439 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.439 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G1Lw+bKKm77x4xbw0lI4q+DzM+8= 2023-09-29T22:19:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G1Lw+bKKm77x4xbw0lI4q+DzM+8=', 'Content-Length': '0'} 2023-09-29T22:19:23.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000872a188e9cbe5ae0-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:23.442 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.442 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.442 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.442 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.442 DEBUG:boto:Method: GET 2023-09-29T22:19:23.442 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.443 DEBUG:boto:Data: 2023-09-29T22:19:23.443 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.443 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.443 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.443 DEBUG:boto:Params: {} 2023-09-29T22:19:23.443 DEBUG:boto:Token: None 2023-09-29T22:19:23.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.443 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021c7c4d7fd7a1462-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.490 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.491 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:23.491 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.491 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.491 DEBUG:boto:Method: GET 2023-09-29T22:19:23.491 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.491 DEBUG:boto:Data: 2023-09-29T22:19:23.491 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.491 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.492 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.492 DEBUG:boto:Params: {} 2023-09-29T22:19:23.492 DEBUG:boto:Token: None 2023-09-29T22:19:23.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:23.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM= 2023-09-29T22:19:23.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yG08WHJVYuTGZeyEwxabksz3MtM=', 'Content-Length': '0'} 2023-09-29T22:19:23.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006a08ac8916cffe7-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.534 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.535 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.535 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.535 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.535 DEBUG:boto:Method: GET 2023-09-29T22:19:23.535 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.535 DEBUG:boto:Data: 2023-09-29T22:19:23.535 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.535 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.535 DEBUG:boto:Params: {} 2023-09-29T22:19:23.536 DEBUG:boto:Token: None 2023-09-29T22:19:23.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0e8e6bfb18d8c8-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.538 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.539 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.539 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.539 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.539 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.539 DEBUG:boto:Method: GET 2023-09-29T22:19:23.539 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:23.540 DEBUG:boto:Data: 2023-09-29T22:19:23.540 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.540 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:23.540 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.540 DEBUG:boto:Params: {} 2023-09-29T22:19:23.540 DEBUG:boto:Token: None 2023-09-29T22:19:23.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b96de272d76641-0065174d6b-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.543 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone3, test-zone1} 2023-09-29T22:19:23.543 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-17 zones={test-zone3, test-zone2} 2023-09-29T22:19:23.543 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.543 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.544 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.544 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:23.544 DEBUG:boto:Data: 2023-09-29T22:19:23.544 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.544 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.544 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.544 DEBUG:boto:Params: {} 2023-09-29T22:19:23.544 DEBUG:boto:Token: None 2023-09-29T22:19:23.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.545 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fiak7nYyXPYs8XwFPDdpeYWZHOU= 2023-09-29T22:19:23.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fiak7nYyXPYs8XwFPDdpeYWZHOU=', 'Content-Length': '0'} 2023-09-29T22:19:23.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f6969d785a48f7f-0065174d6b-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.547 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.547 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.547 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.547 DEBUG:boto:Path: /dmuhsu-17/ 2023-09-29T22:19:23.547 DEBUG:boto:Data: 2023-09-29T22:19:23.547 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.547 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.547 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.547 DEBUG:boto:Params: {} 2023-09-29T22:19:23.548 DEBUG:boto:Token: None 2023-09-29T22:19:23.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.548 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fiak7nYyXPYs8XwFPDdpeYWZHOU= 2023-09-29T22:19:23.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fiak7nYyXPYs8XwFPDdpeYWZHOU=', 'Content-Length': '0'} 2023-09-29T22:19:23.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048764970f9c75e96-0065174d6b-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.550 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:23.551 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.551 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.551 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.551 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.551 DEBUG:boto:Method: GET 2023-09-29T22:19:23.551 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.551 DEBUG:boto:Data: 2023-09-29T22:19:23.551 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.551 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.551 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.552 DEBUG:boto:Params: {} 2023-09-29T22:19:23.552 DEBUG:boto:Token: None 2023-09-29T22:19:23.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/?versions 2023-09-29T22:19:23.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU= 2023-09-29T22:19:23.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU=', 'Content-Length': '0'} 2023-09-29T22:19:23.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef718d44d220ecb0-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.554 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.555 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:23.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:23.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:23.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:23.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:23.555 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.555 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.556 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.556 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.556 DEBUG:boto:Method: GET 2023-09-29T22:19:23.556 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.556 DEBUG:boto:Data: 2023-09-29T22:19:23.556 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.556 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.556 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.556 DEBUG:boto:Params: {} 2023-09-29T22:19:23.556 DEBUG:boto:Token: None 2023-09-29T22:19:23.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/?versions 2023-09-29T22:19:23.557 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU= 2023-09-29T22:19:23.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU=', 'Content-Length': '0'} 2023-09-29T22:19:23.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224089726cfdb761-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.559 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.560 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:23.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:23.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:23.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:23.560 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.560 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.560 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.561 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.561 DEBUG:boto:Method: GET 2023-09-29T22:19:23.561 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.561 DEBUG:boto:Data: 2023-09-29T22:19:23.561 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.561 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.561 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.561 DEBUG:boto:Params: {} 2023-09-29T22:19:23.561 DEBUG:boto:Token: None 2023-09-29T22:19:23.561 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/?versions 2023-09-29T22:19:23.562 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU= 2023-09-29T22:19:23.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU=', 'Content-Length': '0'} 2023-09-29T22:19:23.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa7aa006a759aa16-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.564 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.565 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.565 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.565 DEBUG:boto:path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.565 DEBUG:boto:auth_path=/dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.565 DEBUG:boto:Method: GET 2023-09-29T22:19:23.565 DEBUG:boto:Path: /dmuhsu-17/?versions&max-keys=999 2023-09-29T22:19:23.565 DEBUG:boto:Data: 2023-09-29T22:19:23.565 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.565 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.565 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.566 DEBUG:boto:Params: {} 2023-09-29T22:19:23.566 DEBUG:boto:Token: None 2023-09-29T22:19:23.566 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/?versions 2023-09-29T22:19:23.566 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU= 2023-09-29T22:19:23.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MKWIIpS2bp5XexJEoekAX8eFTaU=', 'Content-Length': '0'} 2023-09-29T22:19:23.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c90d7fd786559982-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.569 DEBUG:boto:b'dmuhsu-17999falsedummyLH4MyYKCflSUJ7VliNTgoWeXAtdOiGLtrue2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gntrue2023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone265f7bMrO9dnfnt84vvkhlOts0eoTcuktrue2023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FIWbFk4igx4ABQsBtdZYK-wNhpKSV6Ntrue2023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:23.569 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.569 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.570 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.570 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.570 DEBUG:boto:Method: GET 2023-09-29T22:19:23.570 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.570 DEBUG:boto:Data: 2023-09-29T22:19:23.570 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.570 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.570 DEBUG:boto:Params: {} 2023-09-29T22:19:23.571 DEBUG:boto:Token: None 2023-09-29T22:19:23.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c5830bb92701222-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.573 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.573 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.573 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.573 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.573 DEBUG:boto:Method: GET 2023-09-29T22:19:23.573 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.573 DEBUG:boto:Data: 2023-09-29T22:19:23.573 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.573 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.573 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.574 DEBUG:boto:Params: {} 2023-09-29T22:19:23.574 DEBUG:boto:Token: None 2023-09-29T22:19:23.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.574 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009316aac71df394f0-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.576 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.576 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.576 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.576 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.576 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.577 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.577 DEBUG:boto:Data: 2023-09-29T22:19:23.577 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.577 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.577 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.577 DEBUG:boto:Params: {} 2023-09-29T22:19:23.577 DEBUG:boto:Token: None 2023-09-29T22:19:23.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.577 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+A8/MXeOeMZ/X/5m5WOQZQE+30= 2023-09-29T22:19:23.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+A8/MXeOeMZ/X/5m5WOQZQE+30=', 'Content-Length': '0'} 2023-09-29T22:19:23.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000440c83dd9319c2ec-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.579 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.579 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.579 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.579 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.580 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.580 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.580 DEBUG:boto:Data: 2023-09-29T22:19:23.580 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.580 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.580 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.580 DEBUG:boto:Params: {} 2023-09-29T22:19:23.580 DEBUG:boto:Token: None 2023-09-29T22:19:23.580 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.581 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w+A8/MXeOeMZ/X/5m5WOQZQE+30= 2023-09-29T22:19:23.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w+A8/MXeOeMZ/X/5m5WOQZQE+30=', 'Content-Length': '0'} 2023-09-29T22:19:23.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000569d58ea8dfd12b9-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:23.583 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.583 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.583 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.583 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.583 DEBUG:boto:Method: GET 2023-09-29T22:19:23.583 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.583 DEBUG:boto:Data: 2023-09-29T22:19:23.583 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.583 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.584 DEBUG:boto:Params: {} 2023-09-29T22:19:23.584 DEBUG:boto:Token: None 2023-09-29T22:19:23.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000887113efcde5a939-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.586 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.586 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.586 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.586 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.586 DEBUG:boto:Method: GET 2023-09-29T22:19:23.586 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.586 DEBUG:boto:Data: 2023-09-29T22:19:23.587 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.587 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.587 DEBUG:boto:Params: {} 2023-09-29T22:19:23.587 DEBUG:boto:Token: None 2023-09-29T22:19:23.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.587 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a1d1ab97117028e-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.589 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.589 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.590 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.590 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.590 DEBUG:boto:Method: GET 2023-09-29T22:19:23.590 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.590 DEBUG:boto:Data: 2023-09-29T22:19:23.590 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.590 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.590 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.590 DEBUG:boto:Params: {} 2023-09-29T22:19:23.590 DEBUG:boto:Token: None 2023-09-29T22:19:23.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9cb56cee2366450-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.638 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.639 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.639 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.639 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.639 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.639 DEBUG:boto:Method: GET 2023-09-29T22:19:23.639 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.639 DEBUG:boto:Data: 2023-09-29T22:19:23.640 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.640 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.640 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.640 DEBUG:boto:Params: {} 2023-09-29T22:19:23.640 DEBUG:boto:Token: None 2023-09-29T22:19:23.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088de28cc3f1a86e6-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.643 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.643 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.643 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.644 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.644 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:23.644 DEBUG:boto:Data: 2023-09-29T22:19:23.644 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.644 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.644 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.644 DEBUG:boto:Params: {} 2023-09-29T22:19:23.644 DEBUG:boto:Token: None 2023-09-29T22:19:23.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy 2023-09-29T22:19:23.645 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MP09JF9rm1ZkLpiWqvxZDTC1FXM= 2023-09-29T22:19:23.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MP09JF9rm1ZkLpiWqvxZDTC1FXM=', 'Content-Length': '0'} 2023-09-29T22:19:23.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e00f275085844d5-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.647 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.647 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.647 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.647 DEBUG:boto:Path: /dmuhsu-17/dummy 2023-09-29T22:19:23.647 DEBUG:boto:Data: 2023-09-29T22:19:23.647 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.647 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.647 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.647 DEBUG:boto:Params: {} 2023-09-29T22:19:23.648 DEBUG:boto:Token: None 2023-09-29T22:19:23.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy 2023-09-29T22:19:23.648 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MP09JF9rm1ZkLpiWqvxZDTC1FXM= 2023-09-29T22:19:23.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MP09JF9rm1ZkLpiWqvxZDTC1FXM=', 'Content-Length': '0'} 2023-09-29T22:19:23.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008405d90527c05bc-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:23.650 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.650 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.651 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.651 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.651 DEBUG:boto:Method: GET 2023-09-29T22:19:23.651 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.651 DEBUG:boto:Data: 2023-09-29T22:19:23.651 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.651 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.651 DEBUG:boto:Params: {} 2023-09-29T22:19:23.652 DEBUG:boto:Token: None 2023-09-29T22:19:23.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005960f2cf7247c0c1-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.654 DEBUG:boto:path=/dmuhsu-17/dummy 2023-09-29T22:19:23.654 DEBUG:boto:auth_path=/dmuhsu-17/dummy 2023-09-29T22:19:23.654 DEBUG:boto:path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.654 DEBUG:boto:auth_path=/dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.654 DEBUG:boto:Method: GET 2023-09-29T22:19:23.654 DEBUG:boto:Path: /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.655 DEBUG:boto:Data: 2023-09-29T22:19:23.655 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.655 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.655 DEBUG:boto:Params: {} 2023-09-29T22:19:23.655 DEBUG:boto:Token: None 2023-09-29T22:19:23.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/dummy?versionId=LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL 2023-09-29T22:19:23.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8= 2023-09-29T22:19:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jP2+ZTNmmUyLmAifWiAV4+6cQW8=', 'Content-Length': '0'} 2023-09-29T22:19:23.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:40 GMT'), ('x-amz-version-id', 'LH4MyYKCflSUJ7VliNTgoWeXAtdOiGL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000384ee5f8267d6063-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.658 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.658 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.658 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.658 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.658 DEBUG:boto:Method: GET 2023-09-29T22:19:23.658 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.658 DEBUG:boto:Data: 2023-09-29T22:19:23.658 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.658 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.658 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.659 DEBUG:boto:Params: {} 2023-09-29T22:19:23.659 DEBUG:boto:Token: None 2023-09-29T22:19:23.659 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.659 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004660c803de54ea4b-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.661 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.662 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.662 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.662 DEBUG:boto:path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.662 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.662 DEBUG:boto:Method: GET 2023-09-29T22:19:23.662 DEBUG:boto:Path: /dmuhsu-17/?prefix=dummy 2023-09-29T22:19:23.662 DEBUG:boto:Data: 2023-09-29T22:19:23.663 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.663 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.663 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.663 DEBUG:boto:Params: {} 2023-09-29T22:19:23.663 DEBUG:boto:Token: None 2023-09-29T22:19:23.663 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.663 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba7d8222221f5be-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.666 DEBUG:boto:b'dmuhsu-17dummy1000falsedummy2023-09-29T22:15:40.751Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:23.666 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.666 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.666 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.667 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.667 DEBUG:boto:Method: GET 2023-09-29T22:19:23.667 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.667 DEBUG:boto:Data: 2023-09-29T22:19:23.667 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.667 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.667 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.667 DEBUG:boto:Params: {} 2023-09-29T22:19:23.667 DEBUG:boto:Token: None 2023-09-29T22:19:23.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de2c3512d7f384df-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.670 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.670 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.670 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.670 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.670 DEBUG:boto:Method: GET 2023-09-29T22:19:23.670 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.670 DEBUG:boto:Data: 2023-09-29T22:19:23.670 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.671 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.671 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.671 DEBUG:boto:Params: {} 2023-09-29T22:19:23.671 DEBUG:boto:Token: None 2023-09-29T22:19:23.671 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.671 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d72f93c48c61b183-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.673 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.674 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.674 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.674 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.674 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.674 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.674 DEBUG:boto:Data: 2023-09-29T22:19:23.674 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.674 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.674 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.675 DEBUG:boto:Params: {} 2023-09-29T22:19:23.675 DEBUG:boto:Token: None 2023-09-29T22:19:23.675 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.675 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DNZaSmvPZLOs5MudV6+IWKhq774= 2023-09-29T22:19:23.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DNZaSmvPZLOs5MudV6+IWKhq774=', 'Content-Length': '0'} 2023-09-29T22:19:23.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000409f4b88b1b3f7d6-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.677 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.677 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.677 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.677 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.677 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.677 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.678 DEBUG:boto:Data: 2023-09-29T22:19:23.678 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.678 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.678 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.678 DEBUG:boto:Params: {} 2023-09-29T22:19:23.678 DEBUG:boto:Token: None 2023-09-29T22:19:23.678 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.678 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DNZaSmvPZLOs5MudV6+IWKhq774= 2023-09-29T22:19:23.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DNZaSmvPZLOs5MudV6+IWKhq774=', 'Content-Length': '0'} 2023-09-29T22:19:23.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abf990c2eb3a5ef5-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:23.681 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.681 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.681 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.681 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.681 DEBUG:boto:Method: GET 2023-09-29T22:19:23.681 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.681 DEBUG:boto:Data: 2023-09-29T22:19:23.681 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.682 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.682 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.682 DEBUG:boto:Params: {} 2023-09-29T22:19:23.682 DEBUG:boto:Token: None 2023-09-29T22:19:23.682 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.682 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9d8e664b69c1a9c-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.726 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.726 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.727 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.727 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.727 DEBUG:boto:Method: GET 2023-09-29T22:19:23.727 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.727 DEBUG:boto:Data: 2023-09-29T22:19:23.727 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.727 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.727 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.727 DEBUG:boto:Params: {} 2023-09-29T22:19:23.728 DEBUG:boto:Token: None 2023-09-29T22:19:23.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df57a3af681b94cf-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.770 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.771 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.771 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.771 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.771 DEBUG:boto:Method: GET 2023-09-29T22:19:23.771 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.771 DEBUG:boto:Data: 2023-09-29T22:19:23.771 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.771 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.771 DEBUG:boto:Params: {} 2023-09-29T22:19:23.772 DEBUG:boto:Token: None 2023-09-29T22:19:23.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037407f3479bd5ea5-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.814 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.815 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.815 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.815 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.815 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.815 DEBUG:boto:Method: GET 2023-09-29T22:19:23.815 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.815 DEBUG:boto:Data: 2023-09-29T22:19:23.816 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.816 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.816 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.816 DEBUG:boto:Params: {} 2023-09-29T22:19:23.816 DEBUG:boto:Token: None 2023-09-29T22:19:23.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004349a11ba0baf010-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.819 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.819 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.819 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.820 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.820 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.820 DEBUG:boto:Data: 2023-09-29T22:19:23.820 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.820 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.820 DEBUG:boto:Params: {} 2023-09-29T22:19:23.820 DEBUG:boto:Token: None 2023-09-29T22:19:23.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.821 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k001IfjRy2tsp5pjB8z7WX+qgrw= 2023-09-29T22:19:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k001IfjRy2tsp5pjB8z7WX+qgrw=', 'Content-Length': '0'} 2023-09-29T22:19:23.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000335d04fef272a6b1-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.823 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.823 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.823 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.823 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.823 DEBUG:boto:Data: 2023-09-29T22:19:23.823 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.824 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.824 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.824 DEBUG:boto:Params: {} 2023-09-29T22:19:23.824 DEBUG:boto:Token: None 2023-09-29T22:19:23.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k001IfjRy2tsp5pjB8z7WX+qgrw= 2023-09-29T22:19:23.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k001IfjRy2tsp5pjB8z7WX+qgrw=', 'Content-Length': '0'} 2023-09-29T22:19:23.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000012cf1f256cfdaf-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:23.827 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.827 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.827 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.827 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.827 DEBUG:boto:Method: GET 2023-09-29T22:19:23.828 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.828 DEBUG:boto:Data: 2023-09-29T22:19:23.828 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.828 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.828 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.828 DEBUG:boto:Params: {} 2023-09-29T22:19:23.828 DEBUG:boto:Token: None 2023-09-29T22:19:23.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004823462f671de8d2-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.874 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.874 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1 2023-09-29T22:19:23.875 DEBUG:boto:path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.875 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.875 DEBUG:boto:Method: GET 2023-09-29T22:19:23.875 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.875 DEBUG:boto:Data: 2023-09-29T22:19:23.875 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.875 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.875 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.876 DEBUG:boto:Params: {} 2023-09-29T22:19:23.876 DEBUG:boto:Token: None 2023-09-29T22:19:23.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone1?versionId=ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn 2023-09-29T22:19:23.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0= 2023-09-29T22:19:23.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7F+CZOfOUAzK146zPCEyCrwuzK0=', 'Content-Length': '0'} 2023-09-29T22:19:23.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ojblwk1Wak9.zS7W0odZX6NDr8Ag7Gn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063f55665f022989c-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.918 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.919 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.919 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.919 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.919 DEBUG:boto:Method: GET 2023-09-29T22:19:23.920 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.920 DEBUG:boto:Data: 2023-09-29T22:19:23.920 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.920 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.920 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.921 DEBUG:boto:Params: {} 2023-09-29T22:19:23.921 DEBUG:boto:Token: None 2023-09-29T22:19:23.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.922 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447a4adfdfab4ff0-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.966 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.967 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:23.967 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:23.967 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.968 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.968 DEBUG:boto:Method: GET 2023-09-29T22:19:23.968 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone1 2023-09-29T22:19:23.968 DEBUG:boto:Data: 2023-09-29T22:19:23.968 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.969 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.969 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.969 DEBUG:boto:Params: {} 2023-09-29T22:19:23.969 DEBUG:boto:Token: None 2023-09-29T22:19:23.970 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/ 2023-09-29T22:19:23.970 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk= 2023-09-29T22:19:23.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pz4Ap5aKWJVwxJVah4kprxbrudk=', 'Content-Length': '0'} 2023-09-29T22:19:23.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f533ef887499e49b-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.973 DEBUG:boto:b'dmuhsu-17obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:23.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:23.974 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.974 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.974 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.974 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.974 DEBUG:boto:Method: GET 2023-09-29T22:19:23.974 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.974 DEBUG:boto:Data: 2023-09-29T22:19:23.975 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.975 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.975 DEBUG:boto:Params: {} 2023-09-29T22:19:23.975 DEBUG:boto:Token: None 2023-09-29T22:19:23.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.975 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baaff3eb80b0d141-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.977 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.977 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.977 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.977 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.977 DEBUG:boto:Method: GET 2023-09-29T22:19:23.978 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.978 DEBUG:boto:Data: 2023-09-29T22:19:23.978 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.978 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.978 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.978 DEBUG:boto:Params: {} 2023-09-29T22:19:23.978 DEBUG:boto:Token: None 2023-09-29T22:19:23.978 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.979 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b3e700a736378c8-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.981 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.981 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.981 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.981 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.981 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.982 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.982 DEBUG:boto:Data: 2023-09-29T22:19:23.982 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.982 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.982 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.982 DEBUG:boto:Params: {} 2023-09-29T22:19:23.982 DEBUG:boto:Token: None 2023-09-29T22:19:23.982 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.983 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4= 2023-09-29T22:19:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4=', 'Content-Length': '0'} 2023-09-29T22:19:23.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000130a22c2b0b7e8e6-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.985 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.985 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.985 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.985 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.985 DEBUG:boto:Method: HEAD 2023-09-29T22:19:23.985 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.985 DEBUG:boto:Data: 2023-09-29T22:19:23.985 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.985 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:23.985 DEBUG:boto:Port: 8000 2023-09-29T22:19:23.986 DEBUG:boto:Params: {} 2023-09-29T22:19:23.986 DEBUG:boto:Token: None 2023-09-29T22:19:23.986 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.986 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4= 2023-09-29T22:19:23.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tBkLC1layKEzkrFIPNy83bnjTP4=', 'Content-Length': '0'} 2023-09-29T22:19:23.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022e607e3cc88595a-0065174d6b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:23.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:23.988 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.988 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:23.989 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.989 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.989 DEBUG:boto:Method: GET 2023-09-29T22:19:23.989 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.989 DEBUG:boto:Data: 2023-09-29T22:19:23.989 DEBUG:boto:Headers: {} 2023-09-29T22:19:23.989 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:23.989 DEBUG:boto:Port: 8001 2023-09-29T22:19:23.989 DEBUG:boto:Params: {} 2023-09-29T22:19:23.989 DEBUG:boto:Token: None 2023-09-29T22:19:23.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:23 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:23.990 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0= 2023-09-29T22:19:23.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bxSyzx5haeaaDt+beh4FMAFidD0=', 'Content-Length': '0'} 2023-09-29T22:19:23.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c566d815b050d1af-0065174d6b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.034 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.035 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.035 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.035 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.035 DEBUG:boto:Method: GET 2023-09-29T22:19:24.035 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.035 DEBUG:boto:Data: 2023-09-29T22:19:24.035 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.035 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.035 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.035 DEBUG:boto:Params: {} 2023-09-29T22:19:24.036 DEBUG:boto:Token: None 2023-09-29T22:19:24.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.036 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME= 2023-09-29T22:19:24.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME=', 'Content-Length': '0'} 2023-09-29T22:19:24.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086758fe886a1e82b-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.038 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.039 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.039 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.039 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.039 DEBUG:boto:Method: GET 2023-09-29T22:19:24.039 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.039 DEBUG:boto:Data: 2023-09-29T22:19:24.039 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.039 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.039 DEBUG:boto:Params: {} 2023-09-29T22:19:24.040 DEBUG:boto:Token: None 2023-09-29T22:19:24.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd489dbd7ff4311-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.042 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.043 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.043 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.043 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.043 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.043 DEBUG:boto:Method: GET 2023-09-29T22:19:24.043 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.044 DEBUG:boto:Data: 2023-09-29T22:19:24.044 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.044 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.044 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.044 DEBUG:boto:Params: {} 2023-09-29T22:19:24.044 DEBUG:boto:Token: None 2023-09-29T22:19:24.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0142a16976165fc-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.047 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.048 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.048 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.048 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.048 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.048 DEBUG:boto:Data: 2023-09-29T22:19:24.048 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.048 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.049 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.049 DEBUG:boto:Params: {} 2023-09-29T22:19:24.049 DEBUG:boto:Token: None 2023-09-29T22:19:24.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.049 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6BUva1BmPkmB6veN46+oU7oB0DY= 2023-09-29T22:19:24.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6BUva1BmPkmB6veN46+oU7oB0DY=', 'Content-Length': '0'} 2023-09-29T22:19:24.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dbdf9004f559bcd-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.051 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.052 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.052 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.052 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.052 DEBUG:boto:Data: 2023-09-29T22:19:24.052 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.052 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.052 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.052 DEBUG:boto:Params: {} 2023-09-29T22:19:24.052 DEBUG:boto:Token: None 2023-09-29T22:19:24.053 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.053 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6BUva1BmPkmB6veN46+oU7oB0DY= 2023-09-29T22:19:24.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6BUva1BmPkmB6veN46+oU7oB0DY=', 'Content-Length': '0'} 2023-09-29T22:19:24.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e3bbc224e8c7752-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:24.055 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.055 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.055 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.056 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.056 DEBUG:boto:Method: GET 2023-09-29T22:19:24.056 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.056 DEBUG:boto:Data: 2023-09-29T22:19:24.056 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.056 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.056 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.056 DEBUG:boto:Params: {} 2023-09-29T22:19:24.056 DEBUG:boto:Token: None 2023-09-29T22:19:24.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.057 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME= 2023-09-29T22:19:24.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME=', 'Content-Length': '0'} 2023-09-29T22:19:24.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1a9785213918120-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.102 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.103 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2 2023-09-29T22:19:24.103 DEBUG:boto:path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.103 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.103 DEBUG:boto:Method: GET 2023-09-29T22:19:24.103 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.103 DEBUG:boto:Data: 2023-09-29T22:19:24.103 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.103 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.103 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.103 DEBUG:boto:Params: {} 2023-09-29T22:19:24.104 DEBUG:boto:Token: None 2023-09-29T22:19:24.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone2?versionId=65f7bMrO9dnfnt84vvkhlOts0eoTcuk 2023-09-29T22:19:24.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME= 2023-09-29T22:19:24.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NjsxALBkbIlYiZj/MGx9X43sNME=', 'Content-Length': '0'} 2023-09-29T22:19:24.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', '65f7bMrO9dnfnt84vvkhlOts0eoTcuk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e03333a6023e9817-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.106 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.106 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.106 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.106 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.107 DEBUG:boto:Method: GET 2023-09-29T22:19:24.107 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.107 DEBUG:boto:Data: 2023-09-29T22:19:24.107 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.107 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.107 DEBUG:boto:Params: {} 2023-09-29T22:19:24.107 DEBUG:boto:Token: None 2023-09-29T22:19:24.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa84ef7e71383a6-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.110 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.110 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.111 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.111 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.111 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.111 DEBUG:boto:Method: GET 2023-09-29T22:19:24.111 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone2 2023-09-29T22:19:24.111 DEBUG:boto:Data: 2023-09-29T22:19:24.111 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.111 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.111 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.111 DEBUG:boto:Params: {} 2023-09-29T22:19:24.112 DEBUG:boto:Token: None 2023-09-29T22:19:24.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d40f135b036c874-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.114 DEBUG:boto:b'dmuhsu-17obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:24.115 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.115 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.115 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.115 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.115 DEBUG:boto:Method: GET 2023-09-29T22:19:24.115 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.115 DEBUG:boto:Data: 2023-09-29T22:19:24.116 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.116 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.116 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.116 DEBUG:boto:Params: {} 2023-09-29T22:19:24.116 DEBUG:boto:Token: None 2023-09-29T22:19:24.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000877fdc2e18a385d3-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.118 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.118 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.119 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.119 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.119 DEBUG:boto:Method: GET 2023-09-29T22:19:24.119 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.119 DEBUG:boto:Data: 2023-09-29T22:19:24.119 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.119 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.120 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.120 DEBUG:boto:Params: {} 2023-09-29T22:19:24.120 DEBUG:boto:Token: None 2023-09-29T22:19:24.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bcc923877f6f50f-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.166 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.166 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.167 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.167 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.167 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.167 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.167 DEBUG:boto:Data: 2023-09-29T22:19:24.167 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.168 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.168 DEBUG:boto:Params: {} 2023-09-29T22:19:24.168 DEBUG:boto:Token: None 2023-09-29T22:19:24.168 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zRhMmXnY+E7pReAjrWzPa7aDSO8= 2023-09-29T22:19:24.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zRhMmXnY+E7pReAjrWzPa7aDSO8=', 'Content-Length': '0'} 2023-09-29T22:19:24.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079152c6255d7006a-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.174 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.174 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.174 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.174 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.174 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.174 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.174 DEBUG:boto:Data: 2023-09-29T22:19:24.175 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.175 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.175 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.175 DEBUG:boto:Params: {} 2023-09-29T22:19:24.175 DEBUG:boto:Token: None 2023-09-29T22:19:24.175 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zRhMmXnY+E7pReAjrWzPa7aDSO8= 2023-09-29T22:19:24.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zRhMmXnY+E7pReAjrWzPa7aDSO8=', 'Content-Length': '0'} 2023-09-29T22:19:24.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e6b7175261f01da-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:24.178 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.178 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.178 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.178 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.178 DEBUG:boto:Method: GET 2023-09-29T22:19:24.178 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.178 DEBUG:boto:Data: 2023-09-29T22:19:24.178 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.179 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.179 DEBUG:boto:Params: {} 2023-09-29T22:19:24.179 DEBUG:boto:Token: None 2023-09-29T22:19:24.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015a22898bba65f11-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.181 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.182 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.182 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.182 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.182 DEBUG:boto:Method: GET 2023-09-29T22:19:24.182 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.182 DEBUG:boto:Data: 2023-09-29T22:19:24.182 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.182 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.182 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.182 DEBUG:boto:Params: {} 2023-09-29T22:19:24.183 DEBUG:boto:Token: None 2023-09-29T22:19:24.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d5976edc3ba97bd-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.226 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.227 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.227 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.227 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.227 DEBUG:boto:Method: GET 2023-09-29T22:19:24.227 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.227 DEBUG:boto:Data: 2023-09-29T22:19:24.227 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.227 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.227 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.227 DEBUG:boto:Params: {} 2023-09-29T22:19:24.228 DEBUG:boto:Token: None 2023-09-29T22:19:24.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053df3fb1fc21dc14-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.231 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.231 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.231 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.231 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.231 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.231 DEBUG:boto:Method: GET 2023-09-29T22:19:24.232 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.232 DEBUG:boto:Data: 2023-09-29T22:19:24.232 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.232 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.232 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.232 DEBUG:boto:Params: {} 2023-09-29T22:19:24.232 DEBUG:boto:Token: None 2023-09-29T22:19:24.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009eab6e3796f9415-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.235 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.235 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.235 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.236 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.236 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.236 DEBUG:boto:Data: 2023-09-29T22:19:24.236 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.236 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.236 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.236 DEBUG:boto:Params: {} 2023-09-29T22:19:24.236 DEBUG:boto:Token: None 2023-09-29T22:19:24.237 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p3igNfoov6FLqfDVPBuqSYwbDpQ= 2023-09-29T22:19:24.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p3igNfoov6FLqfDVPBuqSYwbDpQ=', 'Content-Length': '0'} 2023-09-29T22:19:24.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cef340744c53fff-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.239 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.239 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.239 DEBUG:boto:Method: HEAD 2023-09-29T22:19:24.239 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.239 DEBUG:boto:Data: 2023-09-29T22:19:24.239 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.239 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.240 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.240 DEBUG:boto:Params: {} 2023-09-29T22:19:24.240 DEBUG:boto:Token: None 2023-09-29T22:19:24.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p3igNfoov6FLqfDVPBuqSYwbDpQ= 2023-09-29T22:19:24.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p3igNfoov6FLqfDVPBuqSYwbDpQ=', 'Content-Length': '0'} 2023-09-29T22:19:24.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de08c3002a5513e2-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:24.243 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.243 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.243 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.243 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.243 DEBUG:boto:Method: GET 2023-09-29T22:19:24.243 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.243 DEBUG:boto:Data: 2023-09-29T22:19:24.243 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.243 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.244 DEBUG:boto:Params: {} 2023-09-29T22:19:24.244 DEBUG:boto:Token: None 2023-09-29T22:19:24.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000174140531cc76a7e-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.286 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.287 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3 2023-09-29T22:19:24.287 DEBUG:boto:path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.287 DEBUG:boto:auth_path=/dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.287 DEBUG:boto:Method: GET 2023-09-29T22:19:24.287 DEBUG:boto:Path: /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.287 DEBUG:boto:Data: 2023-09-29T22:19:24.287 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.287 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.287 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.288 DEBUG:boto:Params: {} 2023-09-29T22:19:24.288 DEBUG:boto:Token: None 2023-09-29T22:19:24.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/obj-test-zone3?versionId=FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N 2023-09-29T22:19:24.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo= 2023-09-29T22:19:24.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zuaqCFx1qTqNC08UKjOGhog6sjo=', 'Content-Length': '0'} 2023-09-29T22:19:24.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'FIWbFk4igx4ABQsBtdZYK-wNhpKSV6N'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdd11f47e177b2b4-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.334 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.334 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.335 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.335 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.335 DEBUG:boto:Method: GET 2023-09-29T22:19:24.335 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.335 DEBUG:boto:Data: 2023-09-29T22:19:24.335 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:24.335 DEBUG:boto:Port: 8001 2023-09-29T22:19:24.335 DEBUG:boto:Params: {} 2023-09-29T22:19:24.335 DEBUG:boto:Token: None 2023-09-29T22:19:24.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff85fb316c48c90b-0065174d6c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.339 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.339 DEBUG:boto:path=/dmuhsu-17/ 2023-09-29T22:19:24.339 DEBUG:boto:auth_path=/dmuhsu-17/ 2023-09-29T22:19:24.339 DEBUG:boto:path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.339 DEBUG:boto:auth_path=/dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.339 DEBUG:boto:Method: GET 2023-09-29T22:19:24.339 DEBUG:boto:Path: /dmuhsu-17/?prefix=obj-test-zone3 2023-09-29T22:19:24.340 DEBUG:boto:Data: 2023-09-29T22:19:24.340 DEBUG:boto:Headers: {} 2023-09-29T22:19:24.340 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:24.340 DEBUG:boto:Port: 8000 2023-09-29T22:19:24.340 DEBUG:boto:Params: {} 2023-09-29T22:19:24.340 DEBUG:boto:Token: None 2023-09-29T22:19:24.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:24 GMT /dmuhsu-17/ 2023-09-29T22:19:24.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY= 2023-09-29T22:19:24.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0oQuQFl0XpQSTmLsMfv0E+E3XY=', 'Content-Length': '0'} 2023-09-29T22:19:24.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca746a2e17ff9b73-0065174d6c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:24.343 DEBUG:boto:b'dmuhsu-17obj-test-zone31000falseobj-test-zone32023-09-29T22:16:16.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:24.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-17 zones={test-zone3, test-zone2} 2023-09-29T22:19:24.343 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-18 2023-09-29T22:19:24.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.344 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:24.402 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:24.403 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:24.440 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:24.440+0000 7f540f936880 1 bucket sync caught up with source: 2023-09-29T22:19:24.441 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.48.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:19:24.441 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.48.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:19:24.441 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:24.440+0000 7f540f936880 0 bucket checkpoint complete 2023-09-29T22:19:24.454 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-18 2023-09-29T22:19:24.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.455 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:24.513 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:24.514 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:24.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.552+0000 7fda58f2c880 1 bucket sync caught up with source: 2023-09-29T22:19:24.552 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.48.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:19:24.552 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.48.6, , 00000000003.7.10, , 00000000007.21.10, , , , , ] 2023-09-29T22:19:24.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.552+0000 7fda58f2c880 0 bucket checkpoint complete 2023-09-29T22:19:24.564 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-18 2023-09-29T22:19:24.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.565 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:24.617 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:24.759 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:24.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.651+0000 7f142932e880 1 bucket sync caught up with source: 2023-09-29T22:19:24.761 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000011.27.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:19:24.761 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000011.27.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:19:24.761 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.651+0000 7f142932e880 0 bucket checkpoint complete 2023-09-29T22:19:24.761 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-18 2023-09-29T22:19:24.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.762 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:24.814 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:24.814 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:24.847 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.848+0000 7f672dba2880 1 bucket sync caught up with source: 2023-09-29T22:19:24.848 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.29.10, , 00000000011.27.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:19:24.848 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.29.10, , 00000000011.27.6, , 00000000007.40.10, , , , , ] 2023-09-29T22:19:24.848 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.848+0000 7f672dba2880 0 bucket checkpoint complete 2023-09-29T22:19:24.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-18 2023-09-29T22:19:24.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.860 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:24.912 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:24.912 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:24.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.947+0000 7f6515d33880 1 bucket sync caught up with source: 2023-09-29T22:19:24.947 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000017.64.6, , , , , ] 2023-09-29T22:19:24.947 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000017.64.6, , , , , ] 2023-09-29T22:19:24.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:24.947+0000 7f6515d33880 0 bucket checkpoint complete 2023-09-29T22:19:24.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-18 2023-09-29T22:19:24.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:24.960 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:25.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:25.018 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:25.055 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:25.054+0000 7fefced28880 1 bucket sync caught up with source: 2023-09-29T22:19:25.055 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.28.10, , 00000000003.11.10, , 00000000017.64.6, , , , , ] 2023-09-29T22:19:25.055 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.28.10, , 00000000003.11.10, , 00000000017.64.6, , , , , ] 2023-09-29T22:19:25.055 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:25.054+0000 7fefced28880 0 bucket checkpoint complete 2023-09-29T22:19:25.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone2, test-zone1} 2023-09-29T22:19:25.067 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.067 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.067 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.068 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:25.068 DEBUG:boto:Data: 2023-09-29T22:19:25.068 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.068 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.068 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.068 DEBUG:boto:Params: {} 2023-09-29T22:19:25.068 DEBUG:boto:Token: None 2023-09-29T22:19:25.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0= 2023-09-29T22:19:25.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0=', 'Content-Length': '0'} 2023-09-29T22:19:25.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026fe5eb97000f96a-0065174d6d-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.071 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.071 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.071 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.071 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:25.071 DEBUG:boto:Data: 2023-09-29T22:19:25.072 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.072 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.072 DEBUG:boto:Params: {} 2023-09-29T22:19:25.072 DEBUG:boto:Token: None 2023-09-29T22:19:25.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0= 2023-09-29T22:19:25.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0=', 'Content-Length': '0'} 2023-09-29T22:19:25.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004658479b822bdc7-0065174d6d-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:25.075 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.075 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.075 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.075 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.075 DEBUG:boto:Method: GET 2023-09-29T22:19:25.075 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.075 DEBUG:boto:Data: 2023-09-29T22:19:25.076 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.076 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.076 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.076 DEBUG:boto:Params: {} 2023-09-29T22:19:25.076 DEBUG:boto:Token: None 2023-09-29T22:19:25.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c6566fb390b5cc-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.078 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.079 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:25.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:25.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:25.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:25.079 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:25.079 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.080 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.080 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.080 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.080 DEBUG:boto:Method: GET 2023-09-29T22:19:25.080 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.080 DEBUG:boto:Data: 2023-09-29T22:19:25.080 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.080 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.080 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.080 DEBUG:boto:Params: {} 2023-09-29T22:19:25.081 DEBUG:boto:Token: None 2023-09-29T22:19:25.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.081 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df4d92b73e83c515-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.083 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.083 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:25.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:25.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:25.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:25.084 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.084 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.084 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.084 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.084 DEBUG:boto:Method: GET 2023-09-29T22:19:25.084 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.084 DEBUG:boto:Data: 2023-09-29T22:19:25.085 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.085 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.085 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.085 DEBUG:boto:Params: {} 2023-09-29T22:19:25.085 DEBUG:boto:Token: None 2023-09-29T22:19:25.085 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.085 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eba78847592dbf06-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.087 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.088 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.088 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.088 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.088 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.088 DEBUG:boto:Method: GET 2023-09-29T22:19:25.088 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.089 DEBUG:boto:Data: 2023-09-29T22:19:25.089 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.089 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.089 DEBUG:boto:Params: {} 2023-09-29T22:19:25.089 DEBUG:boto:Token: None 2023-09-29T22:19:25.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.089 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e39ed116129045-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.092 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:25.092 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.092 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.093 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.093 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.093 DEBUG:boto:Method: GET 2023-09-29T22:19:25.093 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.093 DEBUG:boto:Data: 2023-09-29T22:19:25.093 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.093 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.093 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.093 DEBUG:boto:Params: {} 2023-09-29T22:19:25.094 DEBUG:boto:Token: None 2023-09-29T22:19:25.094 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.094 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001946a026725e5a34-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.096 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.096 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.096 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.096 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.096 DEBUG:boto:Method: GET 2023-09-29T22:19:25.096 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.096 DEBUG:boto:Data: 2023-09-29T22:19:25.096 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.096 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.097 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.097 DEBUG:boto:Params: {} 2023-09-29T22:19:25.097 DEBUG:boto:Token: None 2023-09-29T22:19:25.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.097 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e076c092a6e25a39-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.099 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.099 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.099 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.100 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.100 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.100 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.100 DEBUG:boto:Data: 2023-09-29T22:19:25.100 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.100 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.100 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.100 DEBUG:boto:Params: {} 2023-09-29T22:19:25.100 DEBUG:boto:Token: None 2023-09-29T22:19:25.100 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.101 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0= 2023-09-29T22:19:25.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0=', 'Content-Length': '0'} 2023-09-29T22:19:25.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fcfdca790023008-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.102 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.103 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.103 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.103 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.103 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.103 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.103 DEBUG:boto:Data: 2023-09-29T22:19:25.103 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.103 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.103 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.103 DEBUG:boto:Params: {} 2023-09-29T22:19:25.104 DEBUG:boto:Token: None 2023-09-29T22:19:25.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0= 2023-09-29T22:19:25.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0=', 'Content-Length': '0'} 2023-09-29T22:19:25.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e9abde027e63c71-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:25.106 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.106 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.106 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.106 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.106 DEBUG:boto:Method: GET 2023-09-29T22:19:25.106 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.106 DEBUG:boto:Data: 2023-09-29T22:19:25.106 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.107 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.107 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.107 DEBUG:boto:Params: {} 2023-09-29T22:19:25.107 DEBUG:boto:Token: None 2023-09-29T22:19:25.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.107 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e1c7fe1e9ae4fec-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.109 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.109 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.109 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.109 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.109 DEBUG:boto:Method: GET 2023-09-29T22:19:25.110 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.110 DEBUG:boto:Data: 2023-09-29T22:19:25.110 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.110 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.110 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.110 DEBUG:boto:Params: {} 2023-09-29T22:19:25.110 DEBUG:boto:Token: None 2023-09-29T22:19:25.110 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.110 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e96f638ffa293934-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.112 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.112 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.112 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.112 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.113 DEBUG:boto:Method: GET 2023-09-29T22:19:25.113 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.113 DEBUG:boto:Data: 2023-09-29T22:19:25.113 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.113 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.113 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.113 DEBUG:boto:Params: {} 2023-09-29T22:19:25.113 DEBUG:boto:Token: None 2023-09-29T22:19:25.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.114 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c428160936e159e-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.158 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.159 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.159 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.159 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.159 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.159 DEBUG:boto:Method: GET 2023-09-29T22:19:25.159 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.159 DEBUG:boto:Data: 2023-09-29T22:19:25.160 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.160 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.160 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.160 DEBUG:boto:Params: {} 2023-09-29T22:19:25.160 DEBUG:boto:Token: None 2023-09-29T22:19:25.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.160 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e88f9704b46bb3-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.162 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.163 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.163 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.163 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.163 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:25.163 DEBUG:boto:Data: 2023-09-29T22:19:25.163 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.163 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.164 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.164 DEBUG:boto:Params: {} 2023-09-29T22:19:25.164 DEBUG:boto:Token: None 2023-09-29T22:19:25.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy 2023-09-29T22:19:25.164 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vyJ4QeID85s0L5BR+1/ajrfDiZU= 2023-09-29T22:19:25.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vyJ4QeID85s0L5BR+1/ajrfDiZU=', 'Content-Length': '0'} 2023-09-29T22:19:25.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000368ee0b5624a9039-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.166 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.166 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.166 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.167 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:25.167 DEBUG:boto:Data: 2023-09-29T22:19:25.167 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.167 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.167 DEBUG:boto:Params: {} 2023-09-29T22:19:25.167 DEBUG:boto:Token: None 2023-09-29T22:19:25.167 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy 2023-09-29T22:19:25.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vyJ4QeID85s0L5BR+1/ajrfDiZU= 2023-09-29T22:19:25.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vyJ4QeID85s0L5BR+1/ajrfDiZU=', 'Content-Length': '0'} 2023-09-29T22:19:25.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002704ee6acd6e159a-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:25.170 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.170 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.170 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.170 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.171 DEBUG:boto:Method: GET 2023-09-29T22:19:25.171 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.171 DEBUG:boto:Data: 2023-09-29T22:19:25.171 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.171 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.171 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.172 DEBUG:boto:Params: {} 2023-09-29T22:19:25.172 DEBUG:boto:Token: None 2023-09-29T22:19:25.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af6c6fb232832094-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.174 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.174 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.174 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.174 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.175 DEBUG:boto:Method: GET 2023-09-29T22:19:25.175 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.175 DEBUG:boto:Data: 2023-09-29T22:19:25.175 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.175 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.175 DEBUG:boto:Params: {} 2023-09-29T22:19:25.175 DEBUG:boto:Token: None 2023-09-29T22:19:25.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000284ef5ae0e8bbaf0-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.177 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.177 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.178 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.178 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.178 DEBUG:boto:Method: GET 2023-09-29T22:19:25.178 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.178 DEBUG:boto:Data: 2023-09-29T22:19:25.178 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.178 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.178 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.178 DEBUG:boto:Params: {} 2023-09-29T22:19:25.179 DEBUG:boto:Token: None 2023-09-29T22:19:25.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c615069c84795a68-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.181 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.181 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.181 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.182 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.182 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.182 DEBUG:boto:Method: GET 2023-09-29T22:19:25.182 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.182 DEBUG:boto:Data: 2023-09-29T22:19:25.182 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.182 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.182 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.182 DEBUG:boto:Params: {} 2023-09-29T22:19:25.183 DEBUG:boto:Token: None 2023-09-29T22:19:25.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001127c20fcdcc9bdf-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.185 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:25.186 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.186 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.186 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.186 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.186 DEBUG:boto:Method: GET 2023-09-29T22:19:25.186 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.186 DEBUG:boto:Data: 2023-09-29T22:19:25.187 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.187 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.187 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.187 DEBUG:boto:Params: {} 2023-09-29T22:19:25.187 DEBUG:boto:Token: None 2023-09-29T22:19:25.187 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.187 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb229202c8b9919a-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.234 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.235 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.235 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.235 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.235 DEBUG:boto:Method: GET 2023-09-29T22:19:25.235 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.235 DEBUG:boto:Data: 2023-09-29T22:19:25.236 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.236 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.236 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.236 DEBUG:boto:Params: {} 2023-09-29T22:19:25.237 DEBUG:boto:Token: None 2023-09-29T22:19:25.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a1c4be08703336b-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.239 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.239 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.239 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.239 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.240 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.240 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.240 DEBUG:boto:Data: 2023-09-29T22:19:25.240 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.240 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.240 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.240 DEBUG:boto:Params: {} 2023-09-29T22:19:25.240 DEBUG:boto:Token: None 2023-09-29T22:19:25.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.241 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KQlB7hoTSfZQ0hOmyfe4wR1pTCo= 2023-09-29T22:19:25.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KQlB7hoTSfZQ0hOmyfe4wR1pTCo=', 'Content-Length': '0'} 2023-09-29T22:19:25.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f184902b261b774c-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.243 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.243 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.243 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.243 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.243 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.243 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.243 DEBUG:boto:Data: 2023-09-29T22:19:25.243 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.244 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.244 DEBUG:boto:Params: {} 2023-09-29T22:19:25.244 DEBUG:boto:Token: None 2023-09-29T22:19:25.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KQlB7hoTSfZQ0hOmyfe4wR1pTCo= 2023-09-29T22:19:25.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KQlB7hoTSfZQ0hOmyfe4wR1pTCo=', 'Content-Length': '0'} 2023-09-29T22:19:25.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021d530d25f9c38b3-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:25.246 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.246 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.247 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.247 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.247 DEBUG:boto:Method: GET 2023-09-29T22:19:25.247 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.247 DEBUG:boto:Data: 2023-09-29T22:19:25.247 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.247 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.247 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.247 DEBUG:boto:Params: {} 2023-09-29T22:19:25.247 DEBUG:boto:Token: None 2023-09-29T22:19:25.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d21369e4a8df961-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.290 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.291 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.291 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.291 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.291 DEBUG:boto:Method: GET 2023-09-29T22:19:25.291 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.291 DEBUG:boto:Data: 2023-09-29T22:19:25.291 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.291 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.292 DEBUG:boto:Params: {} 2023-09-29T22:19:25.292 DEBUG:boto:Token: None 2023-09-29T22:19:25.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c54f11f0ff0cdd8a-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.294 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.294 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.294 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.294 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.294 DEBUG:boto:Method: GET 2023-09-29T22:19:25.295 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.295 DEBUG:boto:Data: 2023-09-29T22:19:25.295 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.295 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.295 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.295 DEBUG:boto:Params: {} 2023-09-29T22:19:25.295 DEBUG:boto:Token: None 2023-09-29T22:19:25.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ba0276590b24dd-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.338 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.339 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.339 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.339 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.339 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.339 DEBUG:boto:Method: GET 2023-09-29T22:19:25.339 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.340 DEBUG:boto:Data: 2023-09-29T22:19:25.340 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.340 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.340 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.340 DEBUG:boto:Params: {} 2023-09-29T22:19:25.340 DEBUG:boto:Token: None 2023-09-29T22:19:25.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ecd691944322b4-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.343 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.343 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.343 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.343 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.344 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.344 DEBUG:boto:Data: 2023-09-29T22:19:25.344 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.344 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.344 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.344 DEBUG:boto:Params: {} 2023-09-29T22:19:25.344 DEBUG:boto:Token: None 2023-09-29T22:19:25.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jz8nuSbf088ZFzl8n480gIC/HYc= 2023-09-29T22:19:25.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jz8nuSbf088ZFzl8n480gIC/HYc=', 'Content-Length': '0'} 2023-09-29T22:19:25.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008533f357168c3052-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.346 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.347 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.347 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.347 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.347 DEBUG:boto:Data: 2023-09-29T22:19:25.347 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.347 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.347 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.347 DEBUG:boto:Params: {} 2023-09-29T22:19:25.347 DEBUG:boto:Token: None 2023-09-29T22:19:25.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.348 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jz8nuSbf088ZFzl8n480gIC/HYc= 2023-09-29T22:19:25.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jz8nuSbf088ZFzl8n480gIC/HYc=', 'Content-Length': '0'} 2023-09-29T22:19:25.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c827954a9cd9a4d4-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:25.350 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.351 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.351 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.351 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.351 DEBUG:boto:Method: GET 2023-09-29T22:19:25.351 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.351 DEBUG:boto:Data: 2023-09-29T22:19:25.351 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.351 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.351 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.351 DEBUG:boto:Params: {} 2023-09-29T22:19:25.352 DEBUG:boto:Token: None 2023-09-29T22:19:25.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.352 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df51ec8e79bec23f-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.394 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.395 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:25.395 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.395 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.395 DEBUG:boto:Method: GET 2023-09-29T22:19:25.395 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.395 DEBUG:boto:Data: 2023-09-29T22:19:25.395 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.395 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.395 DEBUG:boto:Params: {} 2023-09-29T22:19:25.396 DEBUG:boto:Token: None 2023-09-29T22:19:25.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:25.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY= 2023-09-29T22:19:25.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tY96+ZpyVjSvnmSJ6f3u18smghY=', 'Content-Length': '0'} 2023-09-29T22:19:25.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084989c8dec3bfcf3-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.398 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.398 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.398 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.398 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.398 DEBUG:boto:Method: GET 2023-09-29T22:19:25.398 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.399 DEBUG:boto:Data: 2023-09-29T22:19:25.399 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.399 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.399 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.399 DEBUG:boto:Params: {} 2023-09-29T22:19:25.399 DEBUG:boto:Token: None 2023-09-29T22:19:25.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.399 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001041826fb8af150d-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.402 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.402 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.402 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.402 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.403 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.403 DEBUG:boto:Method: GET 2023-09-29T22:19:25.403 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:25.403 DEBUG:boto:Data: 2023-09-29T22:19:25.403 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.403 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.403 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.403 DEBUG:boto:Params: {} 2023-09-29T22:19:25.403 DEBUG:boto:Token: None 2023-09-29T22:19:25.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9abbb6f4c91ac7e-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.406 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:25.406 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.406 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.406 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.407 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.407 DEBUG:boto:Method: GET 2023-09-29T22:19:25.407 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.407 DEBUG:boto:Data: 2023-09-29T22:19:25.407 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.407 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.407 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.407 DEBUG:boto:Params: {} 2023-09-29T22:19:25.407 DEBUG:boto:Token: None 2023-09-29T22:19:25.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2c73d42a1f3814a-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.411 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.411 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.411 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.411 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.411 DEBUG:boto:Method: GET 2023-09-29T22:19:25.411 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.411 DEBUG:boto:Data: 2023-09-29T22:19:25.411 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.412 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.412 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.412 DEBUG:boto:Params: {} 2023-09-29T22:19:25.412 DEBUG:boto:Token: None 2023-09-29T22:19:25.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006461a6c27c916aee-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.454 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.455 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.455 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.455 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.455 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.455 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.455 DEBUG:boto:Data: 2023-09-29T22:19:25.455 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.455 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.455 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.455 DEBUG:boto:Params: {} 2023-09-29T22:19:25.456 DEBUG:boto:Token: None 2023-09-29T22:19:25.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3il9gZ6XuTdI2crAxcfI5Bf6nfE= 2023-09-29T22:19:25.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3il9gZ6XuTdI2crAxcfI5Bf6nfE=', 'Content-Length': '0'} 2023-09-29T22:19:25.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c4eabb3b09b7f30-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.458 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.458 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.459 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.459 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.459 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.459 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.459 DEBUG:boto:Data: 2023-09-29T22:19:25.459 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.459 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.459 DEBUG:boto:Params: {} 2023-09-29T22:19:25.459 DEBUG:boto:Token: None 2023-09-29T22:19:25.460 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3il9gZ6XuTdI2crAxcfI5Bf6nfE= 2023-09-29T22:19:25.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3il9gZ6XuTdI2crAxcfI5Bf6nfE=', 'Content-Length': '0'} 2023-09-29T22:19:25.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99cdefd6d77fff3-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:25.463 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.463 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.463 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.463 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.463 DEBUG:boto:Method: GET 2023-09-29T22:19:25.463 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.463 DEBUG:boto:Data: 2023-09-29T22:19:25.464 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.464 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.464 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.464 DEBUG:boto:Params: {} 2023-09-29T22:19:25.464 DEBUG:boto:Token: None 2023-09-29T22:19:25.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6217ceda89e5cbf-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.467 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.467 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.467 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.467 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.467 DEBUG:boto:Method: GET 2023-09-29T22:19:25.467 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.467 DEBUG:boto:Data: 2023-09-29T22:19:25.467 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.467 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.467 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.468 DEBUG:boto:Params: {} 2023-09-29T22:19:25.468 DEBUG:boto:Token: None 2023-09-29T22:19:25.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.468 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007207136514fa0766-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.510 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.511 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.511 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.511 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.511 DEBUG:boto:Method: GET 2023-09-29T22:19:25.511 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.511 DEBUG:boto:Data: 2023-09-29T22:19:25.511 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.511 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.512 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.512 DEBUG:boto:Params: {} 2023-09-29T22:19:25.512 DEBUG:boto:Token: None 2023-09-29T22:19:25.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7c88afe1f3fce1-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.515 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.515 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.515 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.516 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.516 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.516 DEBUG:boto:Method: GET 2023-09-29T22:19:25.516 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.516 DEBUG:boto:Data: 2023-09-29T22:19:25.516 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.516 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.516 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.516 DEBUG:boto:Params: {} 2023-09-29T22:19:25.517 DEBUG:boto:Token: None 2023-09-29T22:19:25.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000675a0e007533ffe6-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.519 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.519 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.520 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.520 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.520 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.520 DEBUG:boto:Data: 2023-09-29T22:19:25.520 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.520 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.520 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.520 DEBUG:boto:Params: {} 2023-09-29T22:19:25.520 DEBUG:boto:Token: None 2023-09-29T22:19:25.521 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.521 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gbjzkREcdiOqLmJ094G1oUE2aoU= 2023-09-29T22:19:25.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gbjzkREcdiOqLmJ094G1oUE2aoU=', 'Content-Length': '0'} 2023-09-29T22:19:25.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000219762322d052eb-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.523 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.523 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.523 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.523 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.523 DEBUG:boto:Data: 2023-09-29T22:19:25.524 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.524 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.524 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.524 DEBUG:boto:Params: {} 2023-09-29T22:19:25.524 DEBUG:boto:Token: None 2023-09-29T22:19:25.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gbjzkREcdiOqLmJ094G1oUE2aoU= 2023-09-29T22:19:25.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gbjzkREcdiOqLmJ094G1oUE2aoU=', 'Content-Length': '0'} 2023-09-29T22:19:25.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ea5bddfac47f91-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:25.528 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.528 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.528 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.528 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.529 DEBUG:boto:Method: GET 2023-09-29T22:19:25.529 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.529 DEBUG:boto:Data: 2023-09-29T22:19:25.529 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.529 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.529 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.529 DEBUG:boto:Params: {} 2023-09-29T22:19:25.529 DEBUG:boto:Token: None 2023-09-29T22:19:25.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.530 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca04623fa38e448f-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.574 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.575 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:25.575 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.575 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.575 DEBUG:boto:Method: GET 2023-09-29T22:19:25.575 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.575 DEBUG:boto:Data: 2023-09-29T22:19:25.575 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.575 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.576 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.576 DEBUG:boto:Params: {} 2023-09-29T22:19:25.576 DEBUG:boto:Token: None 2023-09-29T22:19:25.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:25.576 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU= 2023-09-29T22:19:25.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qKP79GzZFe+0hLk/lq1DX44SJHU=', 'Content-Length': '0'} 2023-09-29T22:19:25.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007f9e5392e88b74e-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.619 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.619 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.619 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.619 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.619 DEBUG:boto:Method: GET 2023-09-29T22:19:25.619 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.620 DEBUG:boto:Data: 2023-09-29T22:19:25.620 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.620 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.620 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.620 DEBUG:boto:Params: {} 2023-09-29T22:19:25.620 DEBUG:boto:Token: None 2023-09-29T22:19:25.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.621 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082bad229638e0c7c-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.623 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.623 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.624 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.624 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.624 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.624 DEBUG:boto:Method: GET 2023-09-29T22:19:25.624 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:25.624 DEBUG:boto:Data: 2023-09-29T22:19:25.624 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.624 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.625 DEBUG:boto:Params: {} 2023-09-29T22:19:25.625 DEBUG:boto:Token: None 2023-09-29T22:19:25.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a3f157f3e58738-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.627 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:25.628 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.628 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.628 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.628 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.628 DEBUG:boto:Method: GET 2023-09-29T22:19:25.628 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.628 DEBUG:boto:Data: 2023-09-29T22:19:25.629 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.629 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.629 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.629 DEBUG:boto:Params: {} 2023-09-29T22:19:25.629 DEBUG:boto:Token: None 2023-09-29T22:19:25.629 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.630 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007097aaf6285508d7-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.632 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.632 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.632 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.632 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.632 DEBUG:boto:Method: GET 2023-09-29T22:19:25.632 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.632 DEBUG:boto:Data: 2023-09-29T22:19:25.632 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.632 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.632 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.633 DEBUG:boto:Params: {} 2023-09-29T22:19:25.633 DEBUG:boto:Token: None 2023-09-29T22:19:25.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.633 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000916b440decd736c0-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.635 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.635 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.635 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.635 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.635 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.635 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.636 DEBUG:boto:Data: 2023-09-29T22:19:25.636 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.636 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.636 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.636 DEBUG:boto:Params: {} 2023-09-29T22:19:25.636 DEBUG:boto:Token: None 2023-09-29T22:19:25.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1bIszFO96vFYhDLc9nk3pt+hy9w= 2023-09-29T22:19:25.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1bIszFO96vFYhDLc9nk3pt+hy9w=', 'Content-Length': '0'} 2023-09-29T22:19:25.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ece1c4ee010c7be9-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.638 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.638 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.638 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.639 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.639 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.639 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.639 DEBUG:boto:Data: 2023-09-29T22:19:25.639 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.639 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.639 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.640 DEBUG:boto:Params: {} 2023-09-29T22:19:25.640 DEBUG:boto:Token: None 2023-09-29T22:19:25.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1bIszFO96vFYhDLc9nk3pt+hy9w= 2023-09-29T22:19:25.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1bIszFO96vFYhDLc9nk3pt+hy9w=', 'Content-Length': '0'} 2023-09-29T22:19:25.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042f0372fc5ae8d7c-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:25.642 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.642 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.642 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.642 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.643 DEBUG:boto:Method: GET 2023-09-29T22:19:25.643 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.643 DEBUG:boto:Data: 2023-09-29T22:19:25.643 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.643 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.643 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.643 DEBUG:boto:Params: {} 2023-09-29T22:19:25.643 DEBUG:boto:Token: None 2023-09-29T22:19:25.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d9bc46fa48c2656-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.686 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.687 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.687 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.687 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.687 DEBUG:boto:Method: GET 2023-09-29T22:19:25.687 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.687 DEBUG:boto:Data: 2023-09-29T22:19:25.687 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.687 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.687 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.687 DEBUG:boto:Params: {} 2023-09-29T22:19:25.688 DEBUG:boto:Token: None 2023-09-29T22:19:25.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.688 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399603b4e9436faf-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.734 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.735 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.735 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.735 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.735 DEBUG:boto:Method: GET 2023-09-29T22:19:25.735 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.735 DEBUG:boto:Data: 2023-09-29T22:19:25.735 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.736 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.736 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.736 DEBUG:boto:Params: {} 2023-09-29T22:19:25.736 DEBUG:boto:Token: None 2023-09-29T22:19:25.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ecf6cdf8cad8099-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.739 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.739 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.739 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.739 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.739 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.739 DEBUG:boto:Method: GET 2023-09-29T22:19:25.740 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.740 DEBUG:boto:Data: 2023-09-29T22:19:25.740 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.740 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.740 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.740 DEBUG:boto:Params: {} 2023-09-29T22:19:25.740 DEBUG:boto:Token: None 2023-09-29T22:19:25.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.741 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdc1a8b27351f185-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.743 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.744 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.744 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.744 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.744 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.744 DEBUG:boto:Data: 2023-09-29T22:19:25.744 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.744 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.744 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.745 DEBUG:boto:Params: {} 2023-09-29T22:19:25.745 DEBUG:boto:Token: None 2023-09-29T22:19:25.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s16uEl5GbR4EqJylWEZpQWESNec= 2023-09-29T22:19:25.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s16uEl5GbR4EqJylWEZpQWESNec=', 'Content-Length': '0'} 2023-09-29T22:19:25.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc6fb97f243bb26d-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.747 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.747 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.748 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.748 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.748 DEBUG:boto:Data: 2023-09-29T22:19:25.748 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.748 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.748 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.748 DEBUG:boto:Params: {} 2023-09-29T22:19:25.748 DEBUG:boto:Token: None 2023-09-29T22:19:25.748 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.749 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s16uEl5GbR4EqJylWEZpQWESNec= 2023-09-29T22:19:25.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s16uEl5GbR4EqJylWEZpQWESNec=', 'Content-Length': '0'} 2023-09-29T22:19:25.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075de53d4572bba12-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:25.751 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.751 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.752 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.752 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.752 DEBUG:boto:Method: GET 2023-09-29T22:19:25.752 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.752 DEBUG:boto:Data: 2023-09-29T22:19:25.752 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.752 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.752 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.752 DEBUG:boto:Params: {} 2023-09-29T22:19:25.752 DEBUG:boto:Token: None 2023-09-29T22:19:25.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.753 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b29cd138e06769e0-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.802 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.803 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:25.803 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.803 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.803 DEBUG:boto:Method: GET 2023-09-29T22:19:25.803 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.803 DEBUG:boto:Data: 2023-09-29T22:19:25.803 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.803 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.803 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.803 DEBUG:boto:Params: {} 2023-09-29T22:19:25.804 DEBUG:boto:Token: None 2023-09-29T22:19:25.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:25.804 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E= 2023-09-29T22:19:25.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f7Mjdp2fVYe+NDfuCy3ZOYvzC9E=', 'Content-Length': '0'} 2023-09-29T22:19:25.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046511ac7d42722e3-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.850 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.851 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.851 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.851 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.851 DEBUG:boto:Method: GET 2023-09-29T22:19:25.851 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.851 DEBUG:boto:Data: 2023-09-29T22:19:25.851 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.851 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:25.851 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.851 DEBUG:boto:Params: {} 2023-09-29T22:19:25.852 DEBUG:boto:Token: None 2023-09-29T22:19:25.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088db47bfc51ea44-0065174d6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.854 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.855 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.855 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.855 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.855 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.855 DEBUG:boto:Method: GET 2023-09-29T22:19:25.855 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:25.856 DEBUG:boto:Data: 2023-09-29T22:19:25.856 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.856 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.856 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.856 DEBUG:boto:Params: {} 2023-09-29T22:19:25.856 DEBUG:boto:Token: None 2023-09-29T22:19:25.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a2325b18a9803a-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.902 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone2, test-zone1} 2023-09-29T22:19:25.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone3, test-zone1} 2023-09-29T22:19:25.903 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.903 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.903 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.904 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:25.904 DEBUG:boto:Data: 2023-09-29T22:19:25.904 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.904 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.904 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.904 DEBUG:boto:Params: {} 2023-09-29T22:19:25.904 DEBUG:boto:Token: None 2023-09-29T22:19:25.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0= 2023-09-29T22:19:25.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0=', 'Content-Length': '0'} 2023-09-29T22:19:25.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a1c02472143fe12-0065174d6d-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.907 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.907 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.907 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.907 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:25.907 DEBUG:boto:Data: 2023-09-29T22:19:25.907 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.908 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.908 DEBUG:boto:Params: {} 2023-09-29T22:19:25.908 DEBUG:boto:Token: None 2023-09-29T22:19:25.908 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0= 2023-09-29T22:19:25.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK1evER2diE/304fb+rbv1bX/+0=', 'Content-Length': '0'} 2023-09-29T22:19:25.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007697c093ef6b28ab-0065174d6d-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:25.910 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.910 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.910 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.911 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.911 DEBUG:boto:Method: GET 2023-09-29T22:19:25.911 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.911 DEBUG:boto:Data: 2023-09-29T22:19:25.911 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.911 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.911 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.911 DEBUG:boto:Params: {} 2023-09-29T22:19:25.911 DEBUG:boto:Token: None 2023-09-29T22:19:25.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.912 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd14d5a4515bbe30-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.914 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.914 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:25.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:25.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:25.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:25.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:25.915 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.915 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.915 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.915 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.915 DEBUG:boto:Method: GET 2023-09-29T22:19:25.916 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.916 DEBUG:boto:Data: 2023-09-29T22:19:25.916 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.916 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.916 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.916 DEBUG:boto:Params: {} 2023-09-29T22:19:25.916 DEBUG:boto:Token: None 2023-09-29T22:19:25.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.916 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000617e9f9b7a4861b2-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.919 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.919 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:25.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:25.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:25.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:25.920 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.920 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.920 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.920 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.920 DEBUG:boto:Method: GET 2023-09-29T22:19:25.920 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.920 DEBUG:boto:Data: 2023-09-29T22:19:25.920 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.921 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.921 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.921 DEBUG:boto:Params: {} 2023-09-29T22:19:25.921 DEBUG:boto:Token: None 2023-09-29T22:19:25.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.921 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9716ad9e4b6e8d-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.923 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.924 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.924 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.924 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.924 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.924 DEBUG:boto:Method: GET 2023-09-29T22:19:25.924 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:25.925 DEBUG:boto:Data: 2023-09-29T22:19:25.925 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.925 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.925 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.925 DEBUG:boto:Params: {} 2023-09-29T22:19:25.925 DEBUG:boto:Token: None 2023-09-29T22:19:25.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/?versions 2023-09-29T22:19:25.925 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U= 2023-09-29T22:19:25.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91QdFUMxLk+2tyXtB+O9Ued2r+U=', 'Content-Length': '0'} 2023-09-29T22:19:25.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e69ae5e7ec775b8-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.927 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:25.928 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.928 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.928 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.928 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.929 DEBUG:boto:Method: GET 2023-09-29T22:19:25.929 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.929 DEBUG:boto:Data: 2023-09-29T22:19:25.929 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.929 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.929 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.929 DEBUG:boto:Params: {} 2023-09-29T22:19:25.929 DEBUG:boto:Token: None 2023-09-29T22:19:25.929 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.930 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000aa4441e2257573c-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.932 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.932 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.932 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.932 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.932 DEBUG:boto:Method: GET 2023-09-29T22:19:25.932 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.932 DEBUG:boto:Data: 2023-09-29T22:19:25.932 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.932 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.932 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.933 DEBUG:boto:Params: {} 2023-09-29T22:19:25.933 DEBUG:boto:Token: None 2023-09-29T22:19:25.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000633ce16056eef3c8-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.935 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.935 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.935 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.935 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.935 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.935 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.935 DEBUG:boto:Data: 2023-09-29T22:19:25.935 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.935 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.936 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.936 DEBUG:boto:Params: {} 2023-09-29T22:19:25.936 DEBUG:boto:Token: None 2023-09-29T22:19:25.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0= 2023-09-29T22:19:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0=', 'Content-Length': '0'} 2023-09-29T22:19:25.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b52429edddff2d1-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.938 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.938 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.938 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.938 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.938 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.938 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.938 DEBUG:boto:Data: 2023-09-29T22:19:25.938 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.938 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.939 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.939 DEBUG:boto:Params: {} 2023-09-29T22:19:25.939 DEBUG:boto:Token: None 2023-09-29T22:19:25.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.939 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0= 2023-09-29T22:19:25.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q0/DL4+djVSw2nysmVCAWXHRtp0=', 'Content-Length': '0'} 2023-09-29T22:19:25.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070491b12ddfe47e9-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:25.941 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.941 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.941 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.941 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.941 DEBUG:boto:Method: GET 2023-09-29T22:19:25.941 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.942 DEBUG:boto:Data: 2023-09-29T22:19:25.942 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.942 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.942 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.942 DEBUG:boto:Params: {} 2023-09-29T22:19:25.942 DEBUG:boto:Token: None 2023-09-29T22:19:25.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.942 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005775b21282265127-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.944 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.944 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.944 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.944 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.944 DEBUG:boto:Method: GET 2023-09-29T22:19:25.944 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.945 DEBUG:boto:Data: 2023-09-29T22:19:25.945 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.945 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.945 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.945 DEBUG:boto:Params: {} 2023-09-29T22:19:25.945 DEBUG:boto:Token: None 2023-09-29T22:19:25.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:25.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c= 2023-09-29T22:19:25.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOKqceDaSR1NWEZdfqXAsPTZZ/c=', 'Content-Length': '0'} 2023-09-29T22:19:25.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7a8356129f31c42-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.947 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.947 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.947 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.948 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.948 DEBUG:boto:Method: GET 2023-09-29T22:19:25.948 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.948 DEBUG:boto:Data: 2023-09-29T22:19:25.948 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:25.948 DEBUG:boto:Port: 8001 2023-09-29T22:19:25.948 DEBUG:boto:Params: {} 2023-09-29T22:19:25.948 DEBUG:boto:Token: None 2023-09-29T22:19:25.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075bef03cbe2269a7-0065174d6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.994 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.995 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:25.995 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:25.995 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.995 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.995 DEBUG:boto:Method: GET 2023-09-29T22:19:25.995 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:25.995 DEBUG:boto:Data: 2023-09-29T22:19:25.996 DEBUG:boto:Headers: {} 2023-09-29T22:19:25.996 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:25.996 DEBUG:boto:Port: 8000 2023-09-29T22:19:25.996 DEBUG:boto:Params: {} 2023-09-29T22:19:25.996 DEBUG:boto:Token: None 2023-09-29T22:19:25.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:25 GMT /dmuhsu-18/ 2023-09-29T22:19:25.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4= 2023-09-29T22:19:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VjXP5IHlrCMddhEYNKka6CvvJr4=', 'Content-Length': '0'} 2023-09-29T22:19:25.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b072e467cc7758-0065174d6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:25.999 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:25.999 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:25.999 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:25.999 DEBUG:boto:Method: HEAD 2023-09-29T22:19:25.999 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:25.999 DEBUG:boto:Data: 2023-09-29T22:19:25.999 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.000 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.000 DEBUG:boto:Params: {} 2023-09-29T22:19:26.000 DEBUG:boto:Token: None 2023-09-29T22:19:26.000 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy 2023-09-29T22:19:26.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE= 2023-09-29T22:19:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE=', 'Content-Length': '0'} 2023-09-29T22:19:26.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb5fbf8848b35668-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.002 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.002 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.003 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.003 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:26.003 DEBUG:boto:Data: 2023-09-29T22:19:26.003 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.003 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.003 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.003 DEBUG:boto:Params: {} 2023-09-29T22:19:26.003 DEBUG:boto:Token: None 2023-09-29T22:19:26.003 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy 2023-09-29T22:19:26.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE= 2023-09-29T22:19:26.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE=', 'Content-Length': '0'} 2023-09-29T22:19:26.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9db59fad875ed53-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:26.006 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.006 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.006 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.006 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.006 DEBUG:boto:Method: GET 2023-09-29T22:19:26.006 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.006 DEBUG:boto:Data: 2023-09-29T22:19:26.007 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.007 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.007 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.007 DEBUG:boto:Params: {} 2023-09-29T22:19:26.007 DEBUG:boto:Token: None 2023-09-29T22:19:26.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068a7f468329f91a7-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.009 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.009 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.009 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.009 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.009 DEBUG:boto:Method: GET 2023-09-29T22:19:26.010 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.010 DEBUG:boto:Data: 2023-09-29T22:19:26.010 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.010 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.010 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.010 DEBUG:boto:Params: {} 2023-09-29T22:19:26.010 DEBUG:boto:Token: None 2023-09-29T22:19:26.010 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000228e9f85f8dc557a-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.012 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.012 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.012 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.013 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.013 DEBUG:boto:Method: GET 2023-09-29T22:19:26.013 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.013 DEBUG:boto:Data: 2023-09-29T22:19:26.013 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.013 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.013 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.013 DEBUG:boto:Params: {} 2023-09-29T22:19:26.013 DEBUG:boto:Token: None 2023-09-29T22:19:26.014 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.014 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca63040f2bb6396b-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.016 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.016 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.016 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.017 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.017 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.017 DEBUG:boto:Method: GET 2023-09-29T22:19:26.017 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.017 DEBUG:boto:Data: 2023-09-29T22:19:26.017 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.017 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.017 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.017 DEBUG:boto:Params: {} 2023-09-29T22:19:26.017 DEBUG:boto:Token: None 2023-09-29T22:19:26.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.018 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3984bc978d15ddd-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.020 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.021 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.021 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.021 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.021 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.021 DEBUG:boto:Method: GET 2023-09-29T22:19:26.021 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.021 DEBUG:boto:Data: 2023-09-29T22:19:26.021 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.021 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.021 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.022 DEBUG:boto:Params: {} 2023-09-29T22:19:26.022 DEBUG:boto:Token: None 2023-09-29T22:19:26.022 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.022 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d421c12c179ec4cc-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.024 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.024 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.024 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.024 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.024 DEBUG:boto:Method: GET 2023-09-29T22:19:26.024 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.024 DEBUG:boto:Data: 2023-09-29T22:19:26.024 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.024 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.025 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.025 DEBUG:boto:Params: {} 2023-09-29T22:19:26.025 DEBUG:boto:Token: None 2023-09-29T22:19:26.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d634654abafe14e-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.029 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.029 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.030 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.030 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.030 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.030 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.030 DEBUG:boto:Data: 2023-09-29T22:19:26.030 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.030 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.030 DEBUG:boto:Params: {} 2023-09-29T22:19:26.030 DEBUG:boto:Token: None 2023-09-29T22:19:26.031 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.031 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU= 2023-09-29T22:19:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU=', 'Content-Length': '0'} 2023-09-29T22:19:26.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7726f2715f8e639-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.033 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.033 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.033 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.033 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.033 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.033 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.033 DEBUG:boto:Data: 2023-09-29T22:19:26.033 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.033 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.034 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.034 DEBUG:boto:Params: {} 2023-09-29T22:19:26.034 DEBUG:boto:Token: None 2023-09-29T22:19:26.034 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU= 2023-09-29T22:19:26.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU=', 'Content-Length': '0'} 2023-09-29T22:19:26.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cb51ba1b97faafd-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.036 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.036 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.036 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.036 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.037 DEBUG:boto:Method: GET 2023-09-29T22:19:26.037 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.037 DEBUG:boto:Data: 2023-09-29T22:19:26.037 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.037 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.037 DEBUG:boto:Params: {} 2023-09-29T22:19:26.037 DEBUG:boto:Token: None 2023-09-29T22:19:26.037 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fefd9f6f7dec8ce-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.082 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.082 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.083 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.083 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.083 DEBUG:boto:Method: GET 2023-09-29T22:19:26.083 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.083 DEBUG:boto:Data: 2023-09-29T22:19:26.083 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.083 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.083 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.083 DEBUG:boto:Params: {} 2023-09-29T22:19:26.084 DEBUG:boto:Token: None 2023-09-29T22:19:26.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980596ff4de4cbd5-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.086 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.086 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.086 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.086 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.086 DEBUG:boto:Method: GET 2023-09-29T22:19:26.086 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.087 DEBUG:boto:Data: 2023-09-29T22:19:26.087 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.087 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.087 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.087 DEBUG:boto:Params: {} 2023-09-29T22:19:26.087 DEBUG:boto:Token: None 2023-09-29T22:19:26.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755d1f6a14001344-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.090 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.090 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.090 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.090 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.090 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.091 DEBUG:boto:Method: GET 2023-09-29T22:19:26.091 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.091 DEBUG:boto:Data: 2023-09-29T22:19:26.091 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.091 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.091 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.091 DEBUG:boto:Params: {} 2023-09-29T22:19:26.091 DEBUG:boto:Token: None 2023-09-29T22:19:26.091 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.092 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008238da6bbc3676b0-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.094 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.094 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.094 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.094 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.094 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.095 DEBUG:boto:Data: 2023-09-29T22:19:26.095 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.095 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.095 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.095 DEBUG:boto:Params: {} 2023-09-29T22:19:26.095 DEBUG:boto:Token: None 2023-09-29T22:19:26.095 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.095 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA= 2023-09-29T22:19:26.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA=', 'Content-Length': '0'} 2023-09-29T22:19:26.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000639fe67b9f2b80a1-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.098 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.098 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.098 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.098 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.098 DEBUG:boto:Data: 2023-09-29T22:19:26.098 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.098 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.098 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.099 DEBUG:boto:Params: {} 2023-09-29T22:19:26.099 DEBUG:boto:Token: None 2023-09-29T22:19:26.099 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.099 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA= 2023-09-29T22:19:26.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA=', 'Content-Length': '0'} 2023-09-29T22:19:26.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c46bcf7bf269ae0-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.102 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.102 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.102 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.102 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.102 DEBUG:boto:Method: GET 2023-09-29T22:19:26.102 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.103 DEBUG:boto:Data: 2023-09-29T22:19:26.103 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.103 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.103 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.103 DEBUG:boto:Params: {} 2023-09-29T22:19:26.103 DEBUG:boto:Token: None 2023-09-29T22:19:26.103 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.103 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c92f9926032415d6-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.146 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.147 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.147 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.147 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.147 DEBUG:boto:Method: GET 2023-09-29T22:19:26.147 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.147 DEBUG:boto:Data: 2023-09-29T22:19:26.147 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.148 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.148 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.148 DEBUG:boto:Params: {} 2023-09-29T22:19:26.148 DEBUG:boto:Token: None 2023-09-29T22:19:26.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c92739bcb7e584ce-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.151 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.151 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.151 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.151 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.151 DEBUG:boto:Method: GET 2023-09-29T22:19:26.151 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.152 DEBUG:boto:Data: 2023-09-29T22:19:26.152 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.152 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.152 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.152 DEBUG:boto:Params: {} 2023-09-29T22:19:26.152 DEBUG:boto:Token: None 2023-09-29T22:19:26.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.152 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee6a0ae72e3f2275-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.198 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.199 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.199 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.199 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.199 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.199 DEBUG:boto:Method: GET 2023-09-29T22:19:26.199 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.199 DEBUG:boto:Data: 2023-09-29T22:19:26.199 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.200 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.200 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.200 DEBUG:boto:Params: {} 2023-09-29T22:19:26.200 DEBUG:boto:Token: None 2023-09-29T22:19:26.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000631bb483d3632669-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.202 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:26.203 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.203 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.203 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.203 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.203 DEBUG:boto:Method: GET 2023-09-29T22:19:26.203 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.203 DEBUG:boto:Data: 2023-09-29T22:19:26.204 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.204 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.204 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.204 DEBUG:boto:Params: {} 2023-09-29T22:19:26.204 DEBUG:boto:Token: None 2023-09-29T22:19:26.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad6d66dd9d45d39b-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.206 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.206 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.206 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.207 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.207 DEBUG:boto:Method: GET 2023-09-29T22:19:26.207 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.207 DEBUG:boto:Data: 2023-09-29T22:19:26.207 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.207 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.207 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.207 DEBUG:boto:Params: {} 2023-09-29T22:19:26.207 DEBUG:boto:Token: None 2023-09-29T22:19:26.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e34ba3a3e89db30a-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.250 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.251 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.251 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.251 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.251 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.251 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.251 DEBUG:boto:Data: 2023-09-29T22:19:26.251 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.251 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.251 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.252 DEBUG:boto:Params: {} 2023-09-29T22:19:26.252 DEBUG:boto:Token: None 2023-09-29T22:19:26.252 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0KB+G4F3Rtom0szs2vhe5VQLeOM= 2023-09-29T22:19:26.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0KB+G4F3Rtom0szs2vhe5VQLeOM=', 'Content-Length': '0'} 2023-09-29T22:19:26.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000086c9229fe30da9-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.254 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.254 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.254 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.254 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.255 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.255 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.255 DEBUG:boto:Data: 2023-09-29T22:19:26.255 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.255 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.255 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.255 DEBUG:boto:Params: {} 2023-09-29T22:19:26.255 DEBUG:boto:Token: None 2023-09-29T22:19:26.255 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0KB+G4F3Rtom0szs2vhe5VQLeOM= 2023-09-29T22:19:26.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0KB+G4F3Rtom0szs2vhe5VQLeOM=', 'Content-Length': '0'} 2023-09-29T22:19:26.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075f1585917fad5f2-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:26.258 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.258 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.258 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.258 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.258 DEBUG:boto:Method: GET 2023-09-29T22:19:26.258 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.258 DEBUG:boto:Data: 2023-09-29T22:19:26.259 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.259 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.259 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.259 DEBUG:boto:Params: {} 2023-09-29T22:19:26.259 DEBUG:boto:Token: None 2023-09-29T22:19:26.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.259 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022fb6b7a44ad2671-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.303 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.303 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.303 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.304 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.304 DEBUG:boto:Method: GET 2023-09-29T22:19:26.304 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.305 DEBUG:boto:Data: 2023-09-29T22:19:26.305 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.305 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.305 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.305 DEBUG:boto:Params: {} 2023-09-29T22:19:26.305 DEBUG:boto:Token: None 2023-09-29T22:19:26.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.306 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003719e9852ffd8ce8-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.350 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.351 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.351 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.351 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.351 DEBUG:boto:Method: GET 2023-09-29T22:19:26.351 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.351 DEBUG:boto:Data: 2023-09-29T22:19:26.351 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.351 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.352 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.352 DEBUG:boto:Params: {} 2023-09-29T22:19:26.352 DEBUG:boto:Token: None 2023-09-29T22:19:26.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.352 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062e70180cc8f6ed-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.355 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.355 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.355 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.355 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.356 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.356 DEBUG:boto:Method: GET 2023-09-29T22:19:26.356 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.356 DEBUG:boto:Data: 2023-09-29T22:19:26.356 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.356 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.356 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.356 DEBUG:boto:Params: {} 2023-09-29T22:19:26.357 DEBUG:boto:Token: None 2023-09-29T22:19:26.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa9bfacda2706417-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.360 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.360 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.360 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.360 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.361 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.361 DEBUG:boto:Data: 2023-09-29T22:19:26.361 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.361 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.361 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.361 DEBUG:boto:Params: {} 2023-09-29T22:19:26.361 DEBUG:boto:Token: None 2023-09-29T22:19:26.362 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.362 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqRSyvVTlYlqthZn2JGMJfG64Zs= 2023-09-29T22:19:26.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqRSyvVTlYlqthZn2JGMJfG64Zs=', 'Content-Length': '0'} 2023-09-29T22:19:26.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c8b8de209f2fe70-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.364 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.364 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.365 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.365 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.365 DEBUG:boto:Data: 2023-09-29T22:19:26.365 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.365 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.365 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.365 DEBUG:boto:Params: {} 2023-09-29T22:19:26.366 DEBUG:boto:Token: None 2023-09-29T22:19:26.366 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.366 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dqRSyvVTlYlqthZn2JGMJfG64Zs= 2023-09-29T22:19:26.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dqRSyvVTlYlqthZn2JGMJfG64Zs=', 'Content-Length': '0'} 2023-09-29T22:19:26.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f36ea729fdb0d19-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:26.368 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.368 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.369 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.369 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.369 DEBUG:boto:Method: GET 2023-09-29T22:19:26.369 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.369 DEBUG:boto:Data: 2023-09-29T22:19:26.369 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.370 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.370 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.370 DEBUG:boto:Params: {} 2023-09-29T22:19:26.370 DEBUG:boto:Token: None 2023-09-29T22:19:26.371 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.371 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cc3353b94ad31a0-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.418 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.419 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:26.419 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.419 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.419 DEBUG:boto:Method: GET 2023-09-29T22:19:26.419 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.419 DEBUG:boto:Data: 2023-09-29T22:19:26.419 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.419 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.419 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.419 DEBUG:boto:Params: {} 2023-09-29T22:19:26.420 DEBUG:boto:Token: None 2023-09-29T22:19:26.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:26.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI= 2023-09-29T22:19:26.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A3IZkEmDMv5iSxTfBoArnbEZOyI=', 'Content-Length': '0'} 2023-09-29T22:19:26.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023cda0481275d4d5-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.462 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.463 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.463 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.463 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.463 DEBUG:boto:Method: GET 2023-09-29T22:19:26.463 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.463 DEBUG:boto:Data: 2023-09-29T22:19:26.463 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.464 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.464 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.464 DEBUG:boto:Params: {} 2023-09-29T22:19:26.464 DEBUG:boto:Token: None 2023-09-29T22:19:26.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c5a0de30724f469-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.467 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.467 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.467 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.468 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.468 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.468 DEBUG:boto:Method: GET 2023-09-29T22:19:26.468 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:26.468 DEBUG:boto:Data: 2023-09-29T22:19:26.468 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.468 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.468 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.468 DEBUG:boto:Params: {} 2023-09-29T22:19:26.468 DEBUG:boto:Token: None 2023-09-29T22:19:26.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6300148da8bd43b-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.471 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:26.472 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.472 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.472 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.472 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.472 DEBUG:boto:Method: GET 2023-09-29T22:19:26.472 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.472 DEBUG:boto:Data: 2023-09-29T22:19:26.472 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.472 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.473 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.473 DEBUG:boto:Params: {} 2023-09-29T22:19:26.473 DEBUG:boto:Token: None 2023-09-29T22:19:26.473 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075316f222c2a87ed-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.475 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.475 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.475 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.475 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.475 DEBUG:boto:Method: GET 2023-09-29T22:19:26.476 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.476 DEBUG:boto:Data: 2023-09-29T22:19:26.476 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.476 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.476 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.476 DEBUG:boto:Params: {} 2023-09-29T22:19:26.476 DEBUG:boto:Token: None 2023-09-29T22:19:26.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004ee924001cd4d35-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.478 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.478 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.479 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.479 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.479 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.479 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.479 DEBUG:boto:Data: 2023-09-29T22:19:26.479 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.479 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.479 DEBUG:boto:Params: {} 2023-09-29T22:19:26.480 DEBUG:boto:Token: None 2023-09-29T22:19:26.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wf8kgihHPoAFTlj2XIMkEGEJH8U= 2023-09-29T22:19:26.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wf8kgihHPoAFTlj2XIMkEGEJH8U=', 'Content-Length': '0'} 2023-09-29T22:19:26.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057558187800127d8-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.482 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.482 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.482 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.482 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.482 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.482 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.482 DEBUG:boto:Data: 2023-09-29T22:19:26.483 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.483 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.483 DEBUG:boto:Params: {} 2023-09-29T22:19:26.483 DEBUG:boto:Token: None 2023-09-29T22:19:26.483 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wf8kgihHPoAFTlj2XIMkEGEJH8U= 2023-09-29T22:19:26.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wf8kgihHPoAFTlj2XIMkEGEJH8U=', 'Content-Length': '0'} 2023-09-29T22:19:26.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039f731d9bff7b5c5-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:26.485 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.485 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.485 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.485 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.486 DEBUG:boto:Method: GET 2023-09-29T22:19:26.486 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.486 DEBUG:boto:Data: 2023-09-29T22:19:26.486 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.486 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.486 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.486 DEBUG:boto:Params: {} 2023-09-29T22:19:26.486 DEBUG:boto:Token: None 2023-09-29T22:19:26.486 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.487 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028cc67e2e43fddc3-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.489 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.489 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.489 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.489 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.489 DEBUG:boto:Method: GET 2023-09-29T22:19:26.489 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.489 DEBUG:boto:Data: 2023-09-29T22:19:26.489 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.489 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.490 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.490 DEBUG:boto:Params: {} 2023-09-29T22:19:26.490 DEBUG:boto:Token: None 2023-09-29T22:19:26.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.490 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a46ac09da76efbdd-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.534 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.535 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.535 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.535 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.535 DEBUG:boto:Method: GET 2023-09-29T22:19:26.535 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.536 DEBUG:boto:Data: 2023-09-29T22:19:26.536 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.536 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.536 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.536 DEBUG:boto:Params: {} 2023-09-29T22:19:26.536 DEBUG:boto:Token: None 2023-09-29T22:19:26.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff8d5edb36713a7-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.539 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.540 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.540 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.540 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.540 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.540 DEBUG:boto:Method: GET 2023-09-29T22:19:26.540 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.540 DEBUG:boto:Data: 2023-09-29T22:19:26.540 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.540 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.541 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.541 DEBUG:boto:Params: {} 2023-09-29T22:19:26.541 DEBUG:boto:Token: None 2023-09-29T22:19:26.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.541 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038fe42729582d93-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.544 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.544 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.544 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.544 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.544 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.544 DEBUG:boto:Data: 2023-09-29T22:19:26.544 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.544 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.545 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.545 DEBUG:boto:Params: {} 2023-09-29T22:19:26.545 DEBUG:boto:Token: None 2023-09-29T22:19:26.545 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.545 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHJpjQWZrkZsNRxzq+3Pu02o2gY= 2023-09-29T22:19:26.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHJpjQWZrkZsNRxzq+3Pu02o2gY=', 'Content-Length': '0'} 2023-09-29T22:19:26.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e13223f840ea06d0-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.547 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.547 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.548 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.548 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.548 DEBUG:boto:Data: 2023-09-29T22:19:26.548 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.548 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.548 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.548 DEBUG:boto:Params: {} 2023-09-29T22:19:26.548 DEBUG:boto:Token: None 2023-09-29T22:19:26.549 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HHJpjQWZrkZsNRxzq+3Pu02o2gY= 2023-09-29T22:19:26.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HHJpjQWZrkZsNRxzq+3Pu02o2gY=', 'Content-Length': '0'} 2023-09-29T22:19:26.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007918f09ac457ea8-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:26.551 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.552 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.552 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.552 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.552 DEBUG:boto:Method: GET 2023-09-29T22:19:26.552 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.552 DEBUG:boto:Data: 2023-09-29T22:19:26.552 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.552 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.553 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.553 DEBUG:boto:Params: {} 2023-09-29T22:19:26.553 DEBUG:boto:Token: None 2023-09-29T22:19:26.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.553 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000858d6395f4da355-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.598 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.599 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:26.599 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.599 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.600 DEBUG:boto:Method: GET 2023-09-29T22:19:26.600 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.600 DEBUG:boto:Data: 2023-09-29T22:19:26.600 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.600 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.600 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.600 DEBUG:boto:Params: {} 2023-09-29T22:19:26.600 DEBUG:boto:Token: None 2023-09-29T22:19:26.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:26.601 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4= 2023-09-29T22:19:26.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Oq1gb8LFls4QUeYwlTv4RuSjKR4=', 'Content-Length': '0'} 2023-09-29T22:19:26.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000606dcd358fc15d1e-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.649 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.649 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.649 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.650 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.650 DEBUG:boto:Method: GET 2023-09-29T22:19:26.650 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.650 DEBUG:boto:Data: 2023-09-29T22:19:26.650 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.650 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.650 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.650 DEBUG:boto:Params: {} 2023-09-29T22:19:26.651 DEBUG:boto:Token: None 2023-09-29T22:19:26.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.651 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed176359809d49b3-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.654 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.654 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.654 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.655 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.655 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.655 DEBUG:boto:Method: GET 2023-09-29T22:19:26.655 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:26.655 DEBUG:boto:Data: 2023-09-29T22:19:26.655 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.655 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:26.655 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.655 DEBUG:boto:Params: {} 2023-09-29T22:19:26.656 DEBUG:boto:Token: None 2023-09-29T22:19:26.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077dff4c4c0ff5326-0065174d6e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.698 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone3, test-zone1} 2023-09-29T22:19:26.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-18 zones={test-zone3, test-zone2} 2023-09-29T22:19:26.699 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.699 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.699 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.700 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:26.700 DEBUG:boto:Data: 2023-09-29T22:19:26.700 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.700 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.700 DEBUG:boto:Params: {} 2023-09-29T22:19:26.700 DEBUG:boto:Token: None 2023-09-29T22:19:26.700 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.701 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:knKT77L4oCxKmkfmFkOMzwtzLMs= 2023-09-29T22:19:26.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:knKT77L4oCxKmkfmFkOMzwtzLMs=', 'Content-Length': '0'} 2023-09-29T22:19:26.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f84e65cfe4c7f73-0065174d6e-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.703 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.704 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.704 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.704 DEBUG:boto:Path: /dmuhsu-18/ 2023-09-29T22:19:26.704 DEBUG:boto:Data: 2023-09-29T22:19:26.704 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.704 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.704 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.704 DEBUG:boto:Params: {} 2023-09-29T22:19:26.705 DEBUG:boto:Token: None 2023-09-29T22:19:26.705 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.705 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:knKT77L4oCxKmkfmFkOMzwtzLMs= 2023-09-29T22:19:26.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:knKT77L4oCxKmkfmFkOMzwtzLMs=', 'Content-Length': '0'} 2023-09-29T22:19:26.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eb481b79e8eb646-0065174d6e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.707 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:26.707 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.708 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.708 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.708 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.708 DEBUG:boto:Method: GET 2023-09-29T22:19:26.708 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.708 DEBUG:boto:Data: 2023-09-29T22:19:26.708 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.708 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.708 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.708 DEBUG:boto:Params: {} 2023-09-29T22:19:26.709 DEBUG:boto:Token: None 2023-09-29T22:19:26.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/?versions 2023-09-29T22:19:26.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M= 2023-09-29T22:19:26.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M=', 'Content-Length': '0'} 2023-09-29T22:19:26.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1c80b9c21cd09c-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.712 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.712 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:26.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:26.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:26.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:26.713 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:26.713 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.713 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.713 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.713 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.713 DEBUG:boto:Method: GET 2023-09-29T22:19:26.713 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.713 DEBUG:boto:Data: 2023-09-29T22:19:26.714 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.714 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.714 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.714 DEBUG:boto:Params: {} 2023-09-29T22:19:26.714 DEBUG:boto:Token: None 2023-09-29T22:19:26.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/?versions 2023-09-29T22:19:26.714 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M= 2023-09-29T22:19:26.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M=', 'Content-Length': '0'} 2023-09-29T22:19:26.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac189e54582a9ef-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.717 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.717 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:26.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2023-09-29T22:19:26.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2023-09-29T22:19:26.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2023-09-29T22:19:26.718 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.718 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.718 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.718 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.718 DEBUG:boto:Method: GET 2023-09-29T22:19:26.718 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.719 DEBUG:boto:Data: 2023-09-29T22:19:26.719 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.719 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.719 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.719 DEBUG:boto:Params: {} 2023-09-29T22:19:26.719 DEBUG:boto:Token: None 2023-09-29T22:19:26.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/?versions 2023-09-29T22:19:26.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M= 2023-09-29T22:19:26.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M=', 'Content-Length': '0'} 2023-09-29T22:19:26.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f57eb098cb070b1-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.722 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.722 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.723 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.723 DEBUG:boto:path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.723 DEBUG:boto:auth_path=/dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.723 DEBUG:boto:Method: GET 2023-09-29T22:19:26.723 DEBUG:boto:Path: /dmuhsu-18/?versions&max-keys=999 2023-09-29T22:19:26.723 DEBUG:boto:Data: 2023-09-29T22:19:26.723 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.723 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.723 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.724 DEBUG:boto:Params: {} 2023-09-29T22:19:26.724 DEBUG:boto:Token: None 2023-09-29T22:19:26.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/?versions 2023-09-29T22:19:26.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M= 2023-09-29T22:19:26.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/P/rZms2/2P1wzJjztUdalOvf1M=', 'Content-Length': '0'} 2023-09-29T22:19:26.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007057a5dcd337c45c-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.726 DEBUG:boto:b'dmuhsu-18999falsedummydqwjceWJarNE18U2Lz.E06UPAZac0Gdtrue2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqotrue2023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2ra3FgZs8zAAldybgPyuY7gr0iShfJ7ytrue2023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3obVgoL0C0zabTALO2zLW0QcQb0a.60ztrue2023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:26.727 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.727 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.727 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.727 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.727 DEBUG:boto:Method: GET 2023-09-29T22:19:26.728 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.728 DEBUG:boto:Data: 2023-09-29T22:19:26.728 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.728 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.728 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.728 DEBUG:boto:Params: {} 2023-09-29T22:19:26.728 DEBUG:boto:Token: None 2023-09-29T22:19:26.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f968ca75b7e8ba13-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.730 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.730 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.731 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.731 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.731 DEBUG:boto:Method: GET 2023-09-29T22:19:26.731 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.731 DEBUG:boto:Data: 2023-09-29T22:19:26.731 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.731 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.731 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.731 DEBUG:boto:Params: {} 2023-09-29T22:19:26.731 DEBUG:boto:Token: None 2023-09-29T22:19:26.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6acf9b751cce4b4-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.734 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.734 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.734 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.734 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.734 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.734 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.735 DEBUG:boto:Data: 2023-09-29T22:19:26.735 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.735 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.735 DEBUG:boto:Params: {} 2023-09-29T22:19:26.735 DEBUG:boto:Token: None 2023-09-29T22:19:26.735 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.735 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v61TpEgljJ6/7+LI54CkECL4pJo= 2023-09-29T22:19:26.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v61TpEgljJ6/7+LI54CkECL4pJo=', 'Content-Length': '0'} 2023-09-29T22:19:26.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bfdc752945c8c5c-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.737 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.737 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.737 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.737 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.738 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.738 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.738 DEBUG:boto:Data: 2023-09-29T22:19:26.738 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.738 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.738 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.738 DEBUG:boto:Params: {} 2023-09-29T22:19:26.738 DEBUG:boto:Token: None 2023-09-29T22:19:26.738 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.739 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v61TpEgljJ6/7+LI54CkECL4pJo= 2023-09-29T22:19:26.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v61TpEgljJ6/7+LI54CkECL4pJo=', 'Content-Length': '0'} 2023-09-29T22:19:26.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000653b8660b8f9e979-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:26.741 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.741 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.741 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.741 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.741 DEBUG:boto:Method: GET 2023-09-29T22:19:26.741 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.741 DEBUG:boto:Data: 2023-09-29T22:19:26.741 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.742 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.742 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.742 DEBUG:boto:Params: {} 2023-09-29T22:19:26.742 DEBUG:boto:Token: None 2023-09-29T22:19:26.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000911a1c234df4ad24-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.744 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.744 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.745 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.745 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.745 DEBUG:boto:Method: GET 2023-09-29T22:19:26.745 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.745 DEBUG:boto:Data: 2023-09-29T22:19:26.745 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.745 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.745 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.745 DEBUG:boto:Params: {} 2023-09-29T22:19:26.745 DEBUG:boto:Token: None 2023-09-29T22:19:26.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.746 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff6bc821fc9517f5-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.748 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.748 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.748 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.748 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.748 DEBUG:boto:Method: GET 2023-09-29T22:19:26.749 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.749 DEBUG:boto:Data: 2023-09-29T22:19:26.749 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.749 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.749 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.749 DEBUG:boto:Params: {} 2023-09-29T22:19:26.749 DEBUG:boto:Token: None 2023-09-29T22:19:26.749 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.750 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbfc7ff18c785e52-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.752 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.752 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.753 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.753 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.753 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.753 DEBUG:boto:Method: GET 2023-09-29T22:19:26.753 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.754 DEBUG:boto:Data: 2023-09-29T22:19:26.754 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.754 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.754 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.754 DEBUG:boto:Params: {} 2023-09-29T22:19:26.754 DEBUG:boto:Token: None 2023-09-29T22:19:26.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.754 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b39476b2e1d95a-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.757 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.757 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.757 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.758 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.758 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:26.758 DEBUG:boto:Data: 2023-09-29T22:19:26.758 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.758 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.758 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.758 DEBUG:boto:Params: {} 2023-09-29T22:19:26.758 DEBUG:boto:Token: None 2023-09-29T22:19:26.758 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy 2023-09-29T22:19:26.759 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE= 2023-09-29T22:19:26.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE=', 'Content-Length': '0'} 2023-09-29T22:19:26.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032c1bab14e89832b-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.761 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.761 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.761 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.761 DEBUG:boto:Path: /dmuhsu-18/dummy 2023-09-29T22:19:26.761 DEBUG:boto:Data: 2023-09-29T22:19:26.761 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.761 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.762 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.762 DEBUG:boto:Params: {} 2023-09-29T22:19:26.762 DEBUG:boto:Token: None 2023-09-29T22:19:26.762 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy 2023-09-29T22:19:26.762 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE= 2023-09-29T22:19:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g9D9V2mJTGvyxS1OC70zN9YnpvE=', 'Content-Length': '0'} 2023-09-29T22:19:26.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000408b0b3ecef3845d-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:26.765 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.765 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.765 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.765 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.765 DEBUG:boto:Method: GET 2023-09-29T22:19:26.765 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.765 DEBUG:boto:Data: 2023-09-29T22:19:26.765 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.765 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.766 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.766 DEBUG:boto:Params: {} 2023-09-29T22:19:26.766 DEBUG:boto:Token: None 2023-09-29T22:19:26.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.766 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d03e7f142d1ada9-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.768 DEBUG:boto:path=/dmuhsu-18/dummy 2023-09-29T22:19:26.769 DEBUG:boto:auth_path=/dmuhsu-18/dummy 2023-09-29T22:19:26.769 DEBUG:boto:path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.769 DEBUG:boto:auth_path=/dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.769 DEBUG:boto:Method: GET 2023-09-29T22:19:26.769 DEBUG:boto:Path: /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.769 DEBUG:boto:Data: 2023-09-29T22:19:26.769 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.769 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.769 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.769 DEBUG:boto:Params: {} 2023-09-29T22:19:26.770 DEBUG:boto:Token: None 2023-09-29T22:19:26.770 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/dummy?versionId=dqwjceWJarNE18U2Lz.E06UPAZac0Gd 2023-09-29T22:19:26.770 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o= 2023-09-29T22:19:26.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5ylT6Fif1OKBjbyA3u2wnTS4q0o=', 'Content-Length': '0'} 2023-09-29T22:19:26.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:00 GMT'), ('x-amz-version-id', 'dqwjceWJarNE18U2Lz.E06UPAZac0Gd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080bc60931be17315-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.772 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.772 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.772 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.772 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.772 DEBUG:boto:Method: GET 2023-09-29T22:19:26.773 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.773 DEBUG:boto:Data: 2023-09-29T22:19:26.773 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.773 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.773 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.773 DEBUG:boto:Params: {} 2023-09-29T22:19:26.773 DEBUG:boto:Token: None 2023-09-29T22:19:26.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.773 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f5bba1b80c04043-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.775 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.776 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.776 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.776 DEBUG:boto:path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.776 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.776 DEBUG:boto:Method: GET 2023-09-29T22:19:26.776 DEBUG:boto:Path: /dmuhsu-18/?prefix=dummy 2023-09-29T22:19:26.776 DEBUG:boto:Data: 2023-09-29T22:19:26.777 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.777 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.777 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.777 DEBUG:boto:Params: {} 2023-09-29T22:19:26.777 DEBUG:boto:Token: None 2023-09-29T22:19:26.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.777 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001eebc39b95418c0-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.780 DEBUG:boto:b'dmuhsu-18dummy1000falsedummy2023-09-29T22:16:00.989Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.780 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.780 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.781 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.781 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.781 DEBUG:boto:Method: GET 2023-09-29T22:19:26.781 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.781 DEBUG:boto:Data: 2023-09-29T22:19:26.781 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.781 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.781 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.781 DEBUG:boto:Params: {} 2023-09-29T22:19:26.781 DEBUG:boto:Token: None 2023-09-29T22:19:26.782 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.782 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000633062226e3571cf-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.826 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.826 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.827 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.827 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.827 DEBUG:boto:Method: GET 2023-09-29T22:19:26.827 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.827 DEBUG:boto:Data: 2023-09-29T22:19:26.827 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.827 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.827 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.827 DEBUG:boto:Params: {} 2023-09-29T22:19:26.827 DEBUG:boto:Token: None 2023-09-29T22:19:26.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2096967c7ab73df-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.870 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.871 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.871 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.871 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.871 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.871 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.871 DEBUG:boto:Data: 2023-09-29T22:19:26.871 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.871 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.871 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.871 DEBUG:boto:Params: {} 2023-09-29T22:19:26.872 DEBUG:boto:Token: None 2023-09-29T22:19:26.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU= 2023-09-29T22:19:26.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU=', 'Content-Length': '0'} 2023-09-29T22:19:26.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e90ee9fa800e931a-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.874 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.874 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.874 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.874 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.874 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.875 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.875 DEBUG:boto:Data: 2023-09-29T22:19:26.875 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.875 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.875 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.875 DEBUG:boto:Params: {} 2023-09-29T22:19:26.875 DEBUG:boto:Token: None 2023-09-29T22:19:26.875 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.875 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU= 2023-09-29T22:19:26.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kTQJ7CG/IVj0/D+jlCbkEgKkcOU=', 'Content-Length': '0'} 2023-09-29T22:19:26.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000773f1395bf432deb-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.878 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.878 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.878 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.878 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.878 DEBUG:boto:Method: GET 2023-09-29T22:19:26.878 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.878 DEBUG:boto:Data: 2023-09-29T22:19:26.878 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.878 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.879 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.879 DEBUG:boto:Params: {} 2023-09-29T22:19:26.879 DEBUG:boto:Token: None 2023-09-29T22:19:26.879 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.879 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7a4dc41525fa6ef-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.926 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.927 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.927 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.927 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.927 DEBUG:boto:Method: GET 2023-09-29T22:19:26.927 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.927 DEBUG:boto:Data: 2023-09-29T22:19:26.927 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.927 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.927 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.928 DEBUG:boto:Params: {} 2023-09-29T22:19:26.928 DEBUG:boto:Token: None 2023-09-29T22:19:26.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000add09be6fe170540-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.970 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.971 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.971 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.971 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.971 DEBUG:boto:Method: GET 2023-09-29T22:19:26.971 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.971 DEBUG:boto:Data: 2023-09-29T22:19:26.971 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.972 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.972 DEBUG:boto:Params: {} 2023-09-29T22:19:26.972 DEBUG:boto:Token: None 2023-09-29T22:19:26.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ab0782b2482731b-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.975 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.975 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:26.976 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:26.976 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.976 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.976 DEBUG:boto:Method: GET 2023-09-29T22:19:26.976 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:26.976 DEBUG:boto:Data: 2023-09-29T22:19:26.976 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.976 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.976 DEBUG:boto:Params: {} 2023-09-29T22:19:26.977 DEBUG:boto:Token: None 2023-09-29T22:19:26.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/ 2023-09-29T22:19:26.978 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4= 2023-09-29T22:19:26.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvj3YfKVGnaZ6gr6mbni5Vlt1W4=', 'Content-Length': '0'} 2023-09-29T22:19:26.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c91750150f6483-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.980 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:26.981 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.981 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.981 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.981 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.981 DEBUG:boto:Data: 2023-09-29T22:19:26.981 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.981 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.982 DEBUG:boto:Params: {} 2023-09-29T22:19:26.982 DEBUG:boto:Token: None 2023-09-29T22:19:26.982 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.982 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA= 2023-09-29T22:19:26.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA=', 'Content-Length': '0'} 2023-09-29T22:19:26.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a60dba1d06375ae6-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.985 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.985 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.985 DEBUG:boto:Method: HEAD 2023-09-29T22:19:26.986 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.986 DEBUG:boto:Data: 2023-09-29T22:19:26.986 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.986 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:26.986 DEBUG:boto:Port: 8000 2023-09-29T22:19:26.986 DEBUG:boto:Params: {} 2023-09-29T22:19:26.986 DEBUG:boto:Token: None 2023-09-29T22:19:26.987 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA= 2023-09-29T22:19:26.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgXb5ivjnpC/kCQVYTzsZ6HgpXA=', 'Content-Length': '0'} 2023-09-29T22:19:26.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042b2a9493446951c-0065174d6e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:26.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2023-09-29T22:19:26.989 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.990 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:26.990 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.990 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.990 DEBUG:boto:Method: GET 2023-09-29T22:19:26.990 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.990 DEBUG:boto:Data: 2023-09-29T22:19:26.990 DEBUG:boto:Headers: {} 2023-09-29T22:19:26.990 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:26.991 DEBUG:boto:Port: 8001 2023-09-29T22:19:26.991 DEBUG:boto:Params: {} 2023-09-29T22:19:26.991 DEBUG:boto:Token: None 2023-09-29T22:19:26.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:26 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:26.991 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo= 2023-09-29T22:19:26.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4x10X9M5cWeRMd1CvfeO3PmdPFo=', 'Content-Length': '0'} 2023-09-29T22:19:26.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ff743a6c51865b6-0065174d6e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.038 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:27.039 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1 2023-09-29T22:19:27.039 DEBUG:boto:path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:27.039 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:27.039 DEBUG:boto:Method: GET 2023-09-29T22:19:27.039 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:27.039 DEBUG:boto:Data: 2023-09-29T22:19:27.039 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.039 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.040 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.040 DEBUG:boto:Params: {} 2023-09-29T22:19:27.040 DEBUG:boto:Token: None 2023-09-29T22:19:27.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone1?versionId=pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo 2023-09-29T22:19:27.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wnq1ZQRz1XtnftyaxXQnRCzu73A= 2023-09-29T22:19:27.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wnq1ZQRz1XtnftyaxXQnRCzu73A=', 'Content-Length': '0'} 2023-09-29T22:19:27.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'pvSqqI4K.-i7wPk1hSnhSRjkS5CmXqo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042442033842cf3f4-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.086 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.087 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.087 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.087 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.087 DEBUG:boto:Method: GET 2023-09-29T22:19:27.087 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.087 DEBUG:boto:Data: 2023-09-29T22:19:27.087 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.087 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.088 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.088 DEBUG:boto:Params: {} 2023-09-29T22:19:27.088 DEBUG:boto:Token: None 2023-09-29T22:19:27.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd2cb4a5cf632dd-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.138 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.139 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.139 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.139 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.139 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.139 DEBUG:boto:Method: GET 2023-09-29T22:19:27.140 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone1 2023-09-29T22:19:27.140 DEBUG:boto:Data: 2023-09-29T22:19:27.140 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.140 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.140 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.140 DEBUG:boto:Params: {} 2023-09-29T22:19:27.140 DEBUG:boto:Token: None 2023-09-29T22:19:27.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659ed32425f624e2-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.186 DEBUG:boto:b'dmuhsu-18obj-test-zone11000falseobj-test-zone12023-09-29T22:16:16.804Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:27.187 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.187 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.187 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.187 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.187 DEBUG:boto:Method: GET 2023-09-29T22:19:27.188 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.188 DEBUG:boto:Data: 2023-09-29T22:19:27.188 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.188 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.188 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.188 DEBUG:boto:Params: {} 2023-09-29T22:19:27.188 DEBUG:boto:Token: None 2023-09-29T22:19:27.189 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.189 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f07fecd9fb31cdd7-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.191 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.191 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.191 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.192 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.192 DEBUG:boto:Method: GET 2023-09-29T22:19:27.192 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.192 DEBUG:boto:Data: 2023-09-29T22:19:27.192 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.192 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.192 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.192 DEBUG:boto:Params: {} 2023-09-29T22:19:27.193 DEBUG:boto:Token: None 2023-09-29T22:19:27.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adc15abd2b55b10c-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.195 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.195 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.195 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.196 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.196 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.196 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.196 DEBUG:boto:Data: 2023-09-29T22:19:27.196 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.196 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.196 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.196 DEBUG:boto:Params: {} 2023-09-29T22:19:27.197 DEBUG:boto:Token: None 2023-09-29T22:19:27.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pkkPn/MqmHnhAAlAFQwjDdipXXM= 2023-09-29T22:19:27.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pkkPn/MqmHnhAAlAFQwjDdipXXM=', 'Content-Length': '0'} 2023-09-29T22:19:27.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bf2a129bc9c790d-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.199 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.199 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.200 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.200 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.200 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.200 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.201 DEBUG:boto:Data: 2023-09-29T22:19:27.201 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.201 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.201 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.201 DEBUG:boto:Params: {} 2023-09-29T22:19:27.202 DEBUG:boto:Token: None 2023-09-29T22:19:27.202 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.202 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pkkPn/MqmHnhAAlAFQwjDdipXXM= 2023-09-29T22:19:27.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pkkPn/MqmHnhAAlAFQwjDdipXXM=', 'Content-Length': '0'} 2023-09-29T22:19:27.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005850d254e2ffacc1-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:27.204 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.204 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.204 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.204 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.205 DEBUG:boto:Method: GET 2023-09-29T22:19:27.205 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.205 DEBUG:boto:Data: 2023-09-29T22:19:27.205 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.205 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.205 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.205 DEBUG:boto:Params: {} 2023-09-29T22:19:27.205 DEBUG:boto:Token: None 2023-09-29T22:19:27.206 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.206 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007db993d7ce4e0da4-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.251 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.251 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.251 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.251 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.251 DEBUG:boto:Method: GET 2023-09-29T22:19:27.252 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.252 DEBUG:boto:Data: 2023-09-29T22:19:27.252 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.252 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.252 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.252 DEBUG:boto:Params: {} 2023-09-29T22:19:27.252 DEBUG:boto:Token: None 2023-09-29T22:19:27.253 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.253 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078e74fe150bf3ff1-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.257 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.257 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.257 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.257 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.258 DEBUG:boto:Method: GET 2023-09-29T22:19:27.258 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.258 DEBUG:boto:Data: 2023-09-29T22:19:27.258 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.258 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.258 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.258 DEBUG:boto:Params: {} 2023-09-29T22:19:27.259 DEBUG:boto:Token: None 2023-09-29T22:19:27.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.261 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e19a96f4d1f279b8-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.266 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.268 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.268 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.268 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.269 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.269 DEBUG:boto:Method: GET 2023-09-29T22:19:27.270 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.270 DEBUG:boto:Data: 2023-09-29T22:19:27.270 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.270 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.271 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.271 DEBUG:boto:Params: {} 2023-09-29T22:19:27.271 DEBUG:boto:Token: None 2023-09-29T22:19:27.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583e73c9774ef5db-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.314 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.315 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.315 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.315 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.315 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.315 DEBUG:boto:Data: 2023-09-29T22:19:27.316 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.316 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.316 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.316 DEBUG:boto:Params: {} 2023-09-29T22:19:27.316 DEBUG:boto:Token: None 2023-09-29T22:19:27.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VXVvwLnx5h24JSbK21++8vZTBN4= 2023-09-29T22:19:27.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VXVvwLnx5h24JSbK21++8vZTBN4=', 'Content-Length': '0'} 2023-09-29T22:19:27.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f42622291684a76b-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.319 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.319 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.319 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.319 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.319 DEBUG:boto:Data: 2023-09-29T22:19:27.319 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.320 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.320 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.320 DEBUG:boto:Params: {} 2023-09-29T22:19:27.320 DEBUG:boto:Token: None 2023-09-29T22:19:27.320 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VXVvwLnx5h24JSbK21++8vZTBN4= 2023-09-29T22:19:27.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VXVvwLnx5h24JSbK21++8vZTBN4=', 'Content-Length': '0'} 2023-09-29T22:19:27.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cee31067ff1412c-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2023-09-29T22:19:27.323 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.323 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.323 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.323 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.324 DEBUG:boto:Method: GET 2023-09-29T22:19:27.324 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.324 DEBUG:boto:Data: 2023-09-29T22:19:27.324 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.324 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.324 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.324 DEBUG:boto:Params: {} 2023-09-29T22:19:27.324 DEBUG:boto:Token: None 2023-09-29T22:19:27.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091f05d182f20922c-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.367 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.367 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2 2023-09-29T22:19:27.367 DEBUG:boto:path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.367 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.367 DEBUG:boto:Method: GET 2023-09-29T22:19:27.367 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.367 DEBUG:boto:Data: 2023-09-29T22:19:27.367 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.368 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.368 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.368 DEBUG:boto:Params: {} 2023-09-29T22:19:27.368 DEBUG:boto:Token: None 2023-09-29T22:19:27.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone2?versionId=ra3FgZs8zAAldybgPyuY7gr0iShfJ7y 2023-09-29T22:19:27.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA= 2023-09-29T22:19:27.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:io4/nPryTOppCZMXJpo4aD8/4oA=', 'Content-Length': '0'} 2023-09-29T22:19:27.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:16 GMT'), ('x-amz-version-id', 'ra3FgZs8zAAldybgPyuY7gr0iShfJ7y'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a9a48e383e6cf61-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.371 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.371 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.372 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.372 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.372 DEBUG:boto:Method: GET 2023-09-29T22:19:27.372 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.372 DEBUG:boto:Data: 2023-09-29T22:19:27.372 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.372 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.372 DEBUG:boto:Params: {} 2023-09-29T22:19:27.372 DEBUG:boto:Token: None 2023-09-29T22:19:27.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.373 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0fac5ea5bc6a5f3-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.376 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.376 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.376 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.376 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.376 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.376 DEBUG:boto:Method: GET 2023-09-29T22:19:27.377 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone2 2023-09-29T22:19:27.377 DEBUG:boto:Data: 2023-09-29T22:19:27.377 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.377 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.377 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.377 DEBUG:boto:Params: {} 2023-09-29T22:19:27.377 DEBUG:boto:Token: None 2023-09-29T22:19:27.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.378 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043684ce9db8303bb-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.381 DEBUG:boto:b'dmuhsu-18obj-test-zone21000falseobj-test-zone22023-09-29T22:16:16.901Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:27.381 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.381 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.381 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.381 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.382 DEBUG:boto:Method: GET 2023-09-29T22:19:27.382 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.382 DEBUG:boto:Data: 2023-09-29T22:19:27.382 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.382 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.382 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.382 DEBUG:boto:Params: {} 2023-09-29T22:19:27.382 DEBUG:boto:Token: None 2023-09-29T22:19:27.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e526daed9f468e56-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.386 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.386 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.387 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.387 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.387 DEBUG:boto:Method: GET 2023-09-29T22:19:27.387 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.387 DEBUG:boto:Data: 2023-09-29T22:19:27.387 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.387 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.387 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.387 DEBUG:boto:Params: {} 2023-09-29T22:19:27.387 DEBUG:boto:Token: None 2023-09-29T22:19:27.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046d2527618ff8759-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.390 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.390 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.391 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.391 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.391 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.391 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.391 DEBUG:boto:Data: 2023-09-29T22:19:27.391 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.391 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.391 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.391 DEBUG:boto:Params: {} 2023-09-29T22:19:27.392 DEBUG:boto:Token: None 2023-09-29T22:19:27.392 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BITYCwV4lzy/7G6+CFI1GMSCH3Q= 2023-09-29T22:19:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BITYCwV4lzy/7G6+CFI1GMSCH3Q=', 'Content-Length': '0'} 2023-09-29T22:19:27.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039c49b5d4e568014-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.396 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.396 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.396 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.397 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.397 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.397 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.397 DEBUG:boto:Data: 2023-09-29T22:19:27.397 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.397 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.397 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.397 DEBUG:boto:Params: {} 2023-09-29T22:19:27.397 DEBUG:boto:Token: None 2023-09-29T22:19:27.397 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.398 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BITYCwV4lzy/7G6+CFI1GMSCH3Q= 2023-09-29T22:19:27.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BITYCwV4lzy/7G6+CFI1GMSCH3Q=', 'Content-Length': '0'} 2023-09-29T22:19:27.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec09831918d97479-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:27.400 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.400 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.400 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.400 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.400 DEBUG:boto:Method: GET 2023-09-29T22:19:27.401 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.401 DEBUG:boto:Data: 2023-09-29T22:19:27.401 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.401 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.401 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.401 DEBUG:boto:Params: {} 2023-09-29T22:19:27.401 DEBUG:boto:Token: None 2023-09-29T22:19:27.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.401 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0bfeacacca2822b-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.447 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.447 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.447 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.447 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.448 DEBUG:boto:Method: GET 2023-09-29T22:19:27.448 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.448 DEBUG:boto:Data: 2023-09-29T22:19:27.448 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.448 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.448 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.448 DEBUG:boto:Params: {} 2023-09-29T22:19:27.448 DEBUG:boto:Token: None 2023-09-29T22:19:27.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.449 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cae3739c09eb9fb1-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.495 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.495 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.495 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.495 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.495 DEBUG:boto:Method: GET 2023-09-29T22:19:27.495 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.495 DEBUG:boto:Data: 2023-09-29T22:19:27.496 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.496 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.496 DEBUG:boto:Params: {} 2023-09-29T22:19:27.496 DEBUG:boto:Token: None 2023-09-29T22:19:27.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa253e3d3d9ff058-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.499 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.500 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.500 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.500 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.500 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.500 DEBUG:boto:Method: GET 2023-09-29T22:19:27.500 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.500 DEBUG:boto:Data: 2023-09-29T22:19:27.500 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.501 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.501 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.501 DEBUG:boto:Params: {} 2023-09-29T22:19:27.501 DEBUG:boto:Token: None 2023-09-29T22:19:27.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.501 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c629fd4cc4c4b95d-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.504 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.504 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.504 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.504 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.504 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.504 DEBUG:boto:Data: 2023-09-29T22:19:27.505 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.505 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.505 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.505 DEBUG:boto:Params: {} 2023-09-29T22:19:27.505 DEBUG:boto:Token: None 2023-09-29T22:19:27.505 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KpWmi9uBYY3hOXecpvyft+qhBS0= 2023-09-29T22:19:27.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KpWmi9uBYY3hOXecpvyft+qhBS0=', 'Content-Length': '0'} 2023-09-29T22:19:27.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a031e0cdd48158f-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.508 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.508 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.508 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.508 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.509 DEBUG:boto:Data: 2023-09-29T22:19:27.509 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.509 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.509 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.509 DEBUG:boto:Params: {} 2023-09-29T22:19:27.509 DEBUG:boto:Token: None 2023-09-29T22:19:27.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KpWmi9uBYY3hOXecpvyft+qhBS0= 2023-09-29T22:19:27.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KpWmi9uBYY3hOXecpvyft+qhBS0=', 'Content-Length': '0'} 2023-09-29T22:19:27.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eea808d3791a8e92-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2023-09-29T22:19:27.512 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.512 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.512 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.513 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.513 DEBUG:boto:Method: GET 2023-09-29T22:19:27.513 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.513 DEBUG:boto:Data: 2023-09-29T22:19:27.513 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.513 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.513 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.513 DEBUG:boto:Params: {} 2023-09-29T22:19:27.513 DEBUG:boto:Token: None 2023-09-29T22:19:27.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.514 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a62a0e938da6d1f-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.516 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.516 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3 2023-09-29T22:19:27.516 DEBUG:boto:path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.516 DEBUG:boto:auth_path=/dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.516 DEBUG:boto:Method: GET 2023-09-29T22:19:27.517 DEBUG:boto:Path: /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.517 DEBUG:boto:Data: 2023-09-29T22:19:27.517 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.517 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.517 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.517 DEBUG:boto:Params: {} 2023-09-29T22:19:27.517 DEBUG:boto:Token: None 2023-09-29T22:19:27.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/obj-test-zone3?versionId=obVgoL0C0zabTALO2zLW0QcQb0a.60z 2023-09-29T22:19:27.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo= 2023-09-29T22:19:27.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MdatRAyw2uB44iEkXvhdIxl47Oo=', 'Content-Length': '0'} 2023-09-29T22:19:27.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:16:17 GMT'), ('x-amz-version-id', 'obVgoL0C0zabTALO2zLW0QcQb0a.60z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adc1652eb0b0ca5b-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.562 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.563 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.563 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.563 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.563 DEBUG:boto:Method: GET 2023-09-29T22:19:27.563 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.563 DEBUG:boto:Data: 2023-09-29T22:19:27.563 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.563 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:27.563 DEBUG:boto:Port: 8001 2023-09-29T22:19:27.564 DEBUG:boto:Params: {} 2023-09-29T22:19:27.564 DEBUG:boto:Token: None 2023-09-29T22:19:27.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000134cc9aadf6521e0-0065174d6f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.567 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.567 DEBUG:boto:path=/dmuhsu-18/ 2023-09-29T22:19:27.567 DEBUG:boto:auth_path=/dmuhsu-18/ 2023-09-29T22:19:27.567 DEBUG:boto:path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.567 DEBUG:boto:auth_path=/dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.568 DEBUG:boto:Method: GET 2023-09-29T22:19:27.568 DEBUG:boto:Path: /dmuhsu-18/?prefix=obj-test-zone3 2023-09-29T22:19:27.568 DEBUG:boto:Data: 2023-09-29T22:19:27.568 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.568 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:27.568 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.568 DEBUG:boto:Params: {} 2023-09-29T22:19:27.568 DEBUG:boto:Token: None 2023-09-29T22:19:27.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-18/ 2023-09-29T22:19:27.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4= 2023-09-29T22:19:27.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5cSYZ8GpQMj1I/XVXnjt6NKOtB4=', 'Content-Length': '0'} 2023-09-29T22:19:27.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f1126a621c35b9-0065174d6f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.571 DEBUG:boto:b'dmuhsu-18obj-test-zone31000falseobj-test-zone32023-09-29T22:16:17.003Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:27.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-18 zones={test-zone3, test-zone2} 2023-09-29T22:19:27.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-09-29T22:19:27.574 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:27.574 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:27.574 DEBUG:boto:Method: PUT 2023-09-29T22:19:27.574 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:27.574 DEBUG:boto:Data: 2023-09-29T22:19:27.574 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.574 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:27.575 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.575 DEBUG:boto:Params: {} 2023-09-29T22:19:27.575 DEBUG:boto:Token: None 2023-09-29T22:19:27.575 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-19/ 2023-09-29T22:19:27.575 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LqjMy6uljNyatH0LjqDiSoDQypo= 2023-09-29T22:19:27.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LqjMy6uljNyatH0LjqDiSoDQypo=', 'Content-Length': '0'} 2023-09-29T22:19:27.592 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cefcbd57a9f5e698-0065174d6f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.593 DEBUG:rgw_multi.tests:created bucket=dmuhsu-19 2023-09-29T22:19:27.593 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:27.593 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:27.593 DEBUG:boto:path=/dmuhsu-19/?versioning 2023-09-29T22:19:27.593 DEBUG:boto:auth_path=/dmuhsu-19/?versioning 2023-09-29T22:19:27.593 DEBUG:boto:Method: PUT 2023-09-29T22:19:27.593 DEBUG:boto:Path: /dmuhsu-19/?versioning 2023-09-29T22:19:27.593 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:19:27.593 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.594 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:27.594 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.594 DEBUG:boto:Params: {} 2023-09-29T22:19:27.594 DEBUG:boto:Token: None 2023-09-29T22:19:27.594 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-19/?versioning 2023-09-29T22:19:27.594 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3/ho74QqXC0bwt7hRWBm3KuuSyA= 2023-09-29T22:19:27.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3/ho74QqXC0bwt7hRWBm3KuuSyA=', 'Content-Length': '228'} 2023-09-29T22:19:27.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007216c00d1edbe806-0065174d6f-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.600 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:19:27.651 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:27.651 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:27.682 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:19:27.694 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:19:27.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:19:27.694 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:19:27.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025967.600184_109.1", 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:19:27.600184Z" 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:19:27.696 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:19:27.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.696 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:19:27.697 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025967.600184_109.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:19:27.697 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:19:27.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.697 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:19:27.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:27.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:27.789 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:27.789+0000 7f80f6011880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:19:27.790 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:27.790+0000 7f80f6011880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:19:27.803 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:19:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:19:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:19:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:19:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:19:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025967.600184_109.1", 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:19:27.600184Z", 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:19:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:19:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:19:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:19:27Z" 2023-09-29T22:19:27.810 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:19:27.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025967.600184_109.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:19:27.600184Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:19:27Z" } 2023-09-29T22:19:27.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025967.600184_109.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:19:27.600184Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:19:27.811 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025967.600184_109.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:19:27.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025967.600184_109.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:19:27.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:19:27.811 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:19:27.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:19:27.812 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:19:27.861 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:27.862 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:27.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:27.890+0000 7fbcdd024880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:19:27.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:27.891+0000 7fbcdd024880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:19:27.903 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:19:27.903 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:19:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:19:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:19:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:19:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:19:27.904 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:19:27.905 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:19:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:19:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:19:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:19:27.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:19:27.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025967.600184_109.1", 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:19:27.910 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:19:27.600184Z", 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:19:27.911 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:19:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:19:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:19:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:19:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:19:27.912 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:19:27.913 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:19:27.914 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:19:27Z" 2023-09-29T22:19:27.915 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:19:27.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696025967.600184_109.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:19:27.600184Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:19:27Z" } 2023-09-29T22:19:27.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696025967.600184_109.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:19:27.600184Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:19:27.917 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025967.600184_109.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:19:27.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696025967.600184_109.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:19:27.917 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:19:27.917 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:27.917 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:27.918 DEBUG:boto:Method: HEAD 2023-09-29T22:19:27.918 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:27.918 DEBUG:boto:Data: 2023-09-29T22:19:27.918 DEBUG:boto:Headers: {} 2023-09-29T22:19:27.918 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:27.918 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.918 DEBUG:boto:Params: {} 2023-09-29T22:19:27.919 DEBUG:boto:Token: None 2023-09-29T22:19:27.919 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-19/ 2023-09-29T22:19:27.919 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3HEPOjqY+ekC3LZ8k7H08SwT6nI= 2023-09-29T22:19:27.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3HEPOjqY+ekC3LZ8k7H08SwT6nI=', 'Content-Length': '0'} 2023-09-29T22:19:27.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a59fbc329d7cf52c-0065174d6f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.922 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:27.923 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:27.923 DEBUG:boto:Method: PUT 2023-09-29T22:19:27.923 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:27.923 DEBUG:boto:Data: 2023-09-29T22:19:27.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:19:27.924 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:27.924 DEBUG:boto:Port: 8000 2023-09-29T22:19:27.924 DEBUG:boto:Params: {} 2023-09-29T22:19:27.924 DEBUG:boto:Token: None 2023-09-29T22:19:27.924 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:19:27 GMT /dmuhsu-19/dummy 2023-09-29T22:19:27.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ABZrXU6YbdyAjPv+fd9JWEce4fo= 2023-09-29T22:19:27.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ABZrXU6YbdyAjPv+fd9JWEce4fo='} 2023-09-29T22:19:27.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-amz-request-id', 'tx00000cbec50685f78195e-0065174d6f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:27.936 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-19 2023-09-29T22:19:27.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:27.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:27.936 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:27.993 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:27.994 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:28.030 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:28.029+0000 7fc41e529880 1 waiting to reach incremental sync.. 2023-09-29T22:19:33.030 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:33.030+0000 7fc41e529880 1 waiting to reach incremental sync.. 2023-09-29T22:19:38.032 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:38.031+0000 7fc41e529880 1 bucket sync caught up with source: 2023-09-29T22:19:38.032 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.22.11, , , , , ] 2023-09-29T22:19:38.032 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.22.11, , , , , ] 2023-09-29T22:19:38.032 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:38.031+0000 7fc41e529880 0 bucket checkpoint complete 2023-09-29T22:19:38.047 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-19 2023-09-29T22:19:38.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:38.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:38.048 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:38.105 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:38.106 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:38.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:38.142+0000 7f46f1d23880 1 waiting to reach incremental sync.. 2023-09-29T22:19:43.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.142+0000 7f46f1d23880 1 bucket sync caught up with source: 2023-09-29T22:19:43.144 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:19:43.144 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:19:43.144 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.142+0000 7f46f1d23880 0 bucket checkpoint complete 2023-09-29T22:19:43.158 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-19 2023-09-29T22:19:43.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.159 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:43.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.254+0000 7fa7c6920880 1 bucket sync caught up with source: 2023-09-29T22:19:43.255 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000002.43.11, , , , , ] 2023-09-29T22:19:43.255 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.43.11, , , , , ] 2023-09-29T22:19:43.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.254+0000 7fa7c6920880 0 bucket checkpoint complete 2023-09-29T22:19:43.268 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-19 2023-09-29T22:19:43.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.268 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:43.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:43.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:43.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.353+0000 7f31a44c0880 1 bucket sync caught up with source: 2023-09-29T22:19:43.354 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000002.43.11, , , , , ] 2023-09-29T22:19:43.354 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.43.11, , , , , ] 2023-09-29T22:19:43.354 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.353+0000 7f31a44c0880 0 bucket checkpoint complete 2023-09-29T22:19:43.366 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-19 2023-09-29T22:19:43.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:43.366 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:43.418 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:43.418 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:43.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:43.449+0000 7f284d320880 1 waiting to reach incremental sync.. 2023-09-29T22:19:48.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:48.449+0000 7f284d320880 1 waiting to reach incremental sync.. 2023-09-29T22:19:53.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:53.450+0000 7f284d320880 1 waiting to reach incremental sync.. 2023-09-29T22:19:58.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:58.453+0000 7f284d320880 1 bucket sync caught up with source: 2023-09-29T22:19:58.453 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000003.59.10, , , , , ] 2023-09-29T22:19:58.453 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000003.59.10, , , , , ] 2023-09-29T22:19:58.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:19:58.453+0000 7f284d320880 0 bucket checkpoint complete 2023-09-29T22:19:58.470 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-19 2023-09-29T22:19:58.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:58.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:58.471 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:58.531 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:58.531 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:58.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:58.571+0000 7ffbebbdc880 1 bucket sync caught up with source: 2023-09-29T22:19:58.571 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.59.10, , , , , ] 2023-09-29T22:19:58.571 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000003.59.10, , , , , ] 2023-09-29T22:19:58.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:58.571+0000 7ffbebbdc880 0 bucket checkpoint complete 2023-09-29T22:19:58.584 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone2, test-zone1} 2023-09-29T22:19:58.585 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.585 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.585 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.585 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.585 DEBUG:boto:Data: 2023-09-29T22:19:58.585 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.585 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.585 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.586 DEBUG:boto:Params: {} 2023-09-29T22:19:58.586 DEBUG:boto:Token: None 2023-09-29T22:19:58.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.586 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000475cdf2b270cd339-0065174d8e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.589 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.589 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.590 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.590 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.590 DEBUG:boto:Data: 2023-09-29T22:19:58.590 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.590 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.590 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.590 DEBUG:boto:Params: {} 2023-09-29T22:19:58.591 DEBUG:boto:Token: None 2023-09-29T22:19:58.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2f0b84de7e7f548-0065174d8e-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:58.594 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.594 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.594 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.595 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.595 DEBUG:boto:Method: GET 2023-09-29T22:19:58.595 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.595 DEBUG:boto:Data: 2023-09-29T22:19:58.595 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.595 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.595 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.595 DEBUG:boto:Params: {} 2023-09-29T22:19:58.596 DEBUG:boto:Token: None 2023-09-29T22:19:58.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017a685148c3c4f55-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.598 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.599 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:58.599 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.599 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.599 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.599 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.600 DEBUG:boto:Method: GET 2023-09-29T22:19:58.600 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.600 DEBUG:boto:Data: 2023-09-29T22:19:58.600 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.600 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.600 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.600 DEBUG:boto:Params: {} 2023-09-29T22:19:58.600 DEBUG:boto:Token: None 2023-09-29T22:19:58.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.601 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c366ce921786d97-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.603 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.604 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.604 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.604 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.604 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.604 DEBUG:boto:Method: GET 2023-09-29T22:19:58.604 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.604 DEBUG:boto:Data: 2023-09-29T22:19:58.604 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.605 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.605 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.605 DEBUG:boto:Params: {} 2023-09-29T22:19:58.605 DEBUG:boto:Token: None 2023-09-29T22:19:58.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d5b566a19ba336-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.608 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.609 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.609 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.609 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.609 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.610 DEBUG:boto:Method: GET 2023-09-29T22:19:58.610 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.610 DEBUG:boto:Data: 2023-09-29T22:19:58.610 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.610 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.610 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.610 DEBUG:boto:Params: {} 2023-09-29T22:19:58.610 DEBUG:boto:Token: None 2023-09-29T22:19:58.611 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.611 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bace3f3809fabafd-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.616 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.617 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.617 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.617 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.617 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.617 DEBUG:boto:Method: GET 2023-09-29T22:19:58.617 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.618 DEBUG:boto:Data: 2023-09-29T22:19:58.618 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.618 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.618 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.618 DEBUG:boto:Params: {} 2023-09-29T22:19:58.618 DEBUG:boto:Token: None 2023-09-29T22:19:58.618 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.619 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000364091eab958371-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.621 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.621 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.621 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.621 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.621 DEBUG:boto:Method: GET 2023-09-29T22:19:58.621 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.621 DEBUG:boto:Data: 2023-09-29T22:19:58.621 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.622 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.622 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.622 DEBUG:boto:Params: {} 2023-09-29T22:19:58.622 DEBUG:boto:Token: None 2023-09-29T22:19:58.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.622 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad8b1491d5bd6767-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.625 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.625 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.625 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.625 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.625 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.625 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.625 DEBUG:boto:Data: 2023-09-29T22:19:58.625 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.626 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.626 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.626 DEBUG:boto:Params: {} 2023-09-29T22:19:58.626 DEBUG:boto:Token: None 2023-09-29T22:19:58.626 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.626 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o= 2023-09-29T22:19:58.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o=', 'Content-Length': '0'} 2023-09-29T22:19:58.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f7b72aa455ed3ea-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.629 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.629 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.629 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.629 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.629 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.629 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.629 DEBUG:boto:Data: 2023-09-29T22:19:58.629 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.629 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.630 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.630 DEBUG:boto:Params: {} 2023-09-29T22:19:58.630 DEBUG:boto:Token: None 2023-09-29T22:19:58.630 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.630 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o= 2023-09-29T22:19:58.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o=', 'Content-Length': '0'} 2023-09-29T22:19:58.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f0d09f305dc8a54-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.633 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.633 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.633 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.633 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.633 DEBUG:boto:Method: GET 2023-09-29T22:19:58.634 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.634 DEBUG:boto:Data: 2023-09-29T22:19:58.634 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.634 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.634 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.634 DEBUG:boto:Params: {} 2023-09-29T22:19:58.634 DEBUG:boto:Token: None 2023-09-29T22:19:58.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.635 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000095049c246f3e347-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.637 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.637 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.637 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.638 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.638 DEBUG:boto:Method: GET 2023-09-29T22:19:58.638 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.638 DEBUG:boto:Data: 2023-09-29T22:19:58.638 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.638 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.639 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.639 DEBUG:boto:Params: {} 2023-09-29T22:19:58.639 DEBUG:boto:Token: None 2023-09-29T22:19:58.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.639 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9c223e6ca8105a0-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.642 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.642 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.642 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.642 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.642 DEBUG:boto:Method: GET 2023-09-29T22:19:58.642 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.643 DEBUG:boto:Data: 2023-09-29T22:19:58.643 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.643 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.643 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.643 DEBUG:boto:Params: {} 2023-09-29T22:19:58.643 DEBUG:boto:Token: None 2023-09-29T22:19:58.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3928fc867f4f56f-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.690 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.691 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.691 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.692 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.692 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.692 DEBUG:boto:Method: GET 2023-09-29T22:19:58.692 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.692 DEBUG:boto:Data: 2023-09-29T22:19:58.692 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.692 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.692 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.692 DEBUG:boto:Params: {} 2023-09-29T22:19:58.693 DEBUG:boto:Token: None 2023-09-29T22:19:58.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.693 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb283a842fc1a07-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.696 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.696 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.696 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.697 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.697 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:58.697 DEBUG:boto:Data: 2023-09-29T22:19:58.697 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.697 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.697 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.697 DEBUG:boto:Params: {} 2023-09-29T22:19:58.698 DEBUG:boto:Token: None 2023-09-29T22:19:58.698 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy 2023-09-29T22:19:58.698 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0= 2023-09-29T22:19:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0=', 'Content-Length': '0'} 2023-09-29T22:19:58.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035971492cd33e5c2-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.701 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.701 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.701 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.701 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:58.701 DEBUG:boto:Data: 2023-09-29T22:19:58.701 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.701 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.701 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.702 DEBUG:boto:Params: {} 2023-09-29T22:19:58.702 DEBUG:boto:Token: None 2023-09-29T22:19:58.702 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy 2023-09-29T22:19:58.702 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0= 2023-09-29T22:19:58.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0=', 'Content-Length': '0'} 2023-09-29T22:19:58.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d68de4e5a1290767-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.705 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.706 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.706 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.706 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.706 DEBUG:boto:Method: GET 2023-09-29T22:19:58.707 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.707 DEBUG:boto:Data: 2023-09-29T22:19:58.707 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.707 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.707 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.707 DEBUG:boto:Params: {} 2023-09-29T22:19:58.707 DEBUG:boto:Token: None 2023-09-29T22:19:58.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.708 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000653197d7210a9930-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.710 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.710 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.710 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.711 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.711 DEBUG:boto:Method: GET 2023-09-29T22:19:58.711 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.711 DEBUG:boto:Data: 2023-09-29T22:19:58.711 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.712 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.712 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.712 DEBUG:boto:Params: {} 2023-09-29T22:19:58.712 DEBUG:boto:Token: None 2023-09-29T22:19:58.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000138a6a2d7a5955f-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.715 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.715 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.715 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.716 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.716 DEBUG:boto:Method: GET 2023-09-29T22:19:58.716 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.716 DEBUG:boto:Data: 2023-09-29T22:19:58.716 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.717 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.717 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.717 DEBUG:boto:Params: {} 2023-09-29T22:19:58.717 DEBUG:boto:Token: None 2023-09-29T22:19:58.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.718 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ec302830eb10d5-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.721 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.721 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.722 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.722 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.722 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.722 DEBUG:boto:Method: GET 2023-09-29T22:19:58.723 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.723 DEBUG:boto:Data: 2023-09-29T22:19:58.723 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.723 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.723 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.723 DEBUG:boto:Params: {} 2023-09-29T22:19:58.723 DEBUG:boto:Token: None 2023-09-29T22:19:58.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceacaa85192ebded-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.728 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.728 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone2, test-zone1} 2023-09-29T22:19:58.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone3, test-zone1} 2023-09-29T22:19:58.729 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.729 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.729 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.730 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.730 DEBUG:boto:Data: 2023-09-29T22:19:58.730 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.730 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.730 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.730 DEBUG:boto:Params: {} 2023-09-29T22:19:58.731 DEBUG:boto:Token: None 2023-09-29T22:19:58.731 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.731 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048882b03a5608a81-0065174d8e-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.735 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.735 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.735 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.736 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.736 DEBUG:boto:Data: 2023-09-29T22:19:58.736 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.736 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.736 DEBUG:boto:Params: {} 2023-09-29T22:19:58.737 DEBUG:boto:Token: None 2023-09-29T22:19:58.737 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000319e6dab69eb2c44-0065174d8e-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.740 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:58.740 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.740 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.741 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.741 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.741 DEBUG:boto:Method: GET 2023-09-29T22:19:58.741 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.741 DEBUG:boto:Data: 2023-09-29T22:19:58.742 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.742 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.742 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.742 DEBUG:boto:Params: {} 2023-09-29T22:19:58.742 DEBUG:boto:Token: None 2023-09-29T22:19:58.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.743 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7db66a4caf4926a-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.787 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.787 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:58.787 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.788 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.788 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.788 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.788 DEBUG:boto:Method: GET 2023-09-29T22:19:58.788 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.788 DEBUG:boto:Data: 2023-09-29T22:19:58.788 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.789 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.789 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.789 DEBUG:boto:Params: {} 2023-09-29T22:19:58.789 DEBUG:boto:Token: None 2023-09-29T22:19:58.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.789 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e4d50aef2a3926-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.792 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.793 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.793 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.793 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.793 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.793 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.793 DEBUG:boto:Method: GET 2023-09-29T22:19:58.793 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.793 DEBUG:boto:Data: 2023-09-29T22:19:58.793 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.794 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.794 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.794 DEBUG:boto:Params: {} 2023-09-29T22:19:58.794 DEBUG:boto:Token: None 2023-09-29T22:19:58.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.794 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009786311d77ee2eb-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.797 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.798 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.798 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.798 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.798 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.798 DEBUG:boto:Method: GET 2023-09-29T22:19:58.798 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.799 DEBUG:boto:Data: 2023-09-29T22:19:58.799 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.799 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.799 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.799 DEBUG:boto:Params: {} 2023-09-29T22:19:58.799 DEBUG:boto:Token: None 2023-09-29T22:19:58.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.799 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72c6e8e8f6e9ff3-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.803 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.803 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.804 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.804 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.804 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.804 DEBUG:boto:Method: GET 2023-09-29T22:19:58.804 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.804 DEBUG:boto:Data: 2023-09-29T22:19:58.804 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.804 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.804 DEBUG:boto:Params: {} 2023-09-29T22:19:58.805 DEBUG:boto:Token: None 2023-09-29T22:19:58.805 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f06cb6de21043eeb-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.807 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.807 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.808 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.808 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.808 DEBUG:boto:Method: GET 2023-09-29T22:19:58.808 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.808 DEBUG:boto:Data: 2023-09-29T22:19:58.808 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.808 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.809 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.809 DEBUG:boto:Params: {} 2023-09-29T22:19:58.809 DEBUG:boto:Token: None 2023-09-29T22:19:58.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004376c937494e9ab1-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.812 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.812 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.812 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.812 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.812 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.812 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.812 DEBUG:boto:Data: 2023-09-29T22:19:58.812 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.813 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.813 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.813 DEBUG:boto:Params: {} 2023-09-29T22:19:58.813 DEBUG:boto:Token: None 2023-09-29T22:19:58.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.813 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o= 2023-09-29T22:19:58.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o=', 'Content-Length': '0'} 2023-09-29T22:19:58.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d553a50df7a74b5a-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.815 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.816 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.816 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.816 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.816 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.816 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.816 DEBUG:boto:Data: 2023-09-29T22:19:58.817 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.817 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.817 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.817 DEBUG:boto:Params: {} 2023-09-29T22:19:58.817 DEBUG:boto:Token: None 2023-09-29T22:19:58.817 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o= 2023-09-29T22:19:58.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eb0imGnzpkGTXK9E7WE09cJGv7o=', 'Content-Length': '0'} 2023-09-29T22:19:58.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000464ddb5629bfb791-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.820 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.820 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.820 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.821 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.821 DEBUG:boto:Method: GET 2023-09-29T22:19:58.821 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.821 DEBUG:boto:Data: 2023-09-29T22:19:58.821 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.821 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.822 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.822 DEBUG:boto:Params: {} 2023-09-29T22:19:58.822 DEBUG:boto:Token: None 2023-09-29T22:19:58.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.823 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e24345fbd445a76b-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.825 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.825 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.826 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.826 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.826 DEBUG:boto:Method: GET 2023-09-29T22:19:58.826 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.826 DEBUG:boto:Data: 2023-09-29T22:19:58.826 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.827 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.827 DEBUG:boto:Params: {} 2023-09-29T22:19:58.828 DEBUG:boto:Token: None 2023-09-29T22:19:58.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025eab1f88eea4585-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.831 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.831 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.832 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.832 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.832 DEBUG:boto:Method: GET 2023-09-29T22:19:58.832 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.832 DEBUG:boto:Data: 2023-09-29T22:19:58.832 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.833 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.833 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.833 DEBUG:boto:Params: {} 2023-09-29T22:19:58.833 DEBUG:boto:Token: None 2023-09-29T22:19:58.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.834 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b476b7a4ea4d4afe-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.879 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.879 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.880 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.880 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.880 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.880 DEBUG:boto:Method: GET 2023-09-29T22:19:58.880 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.880 DEBUG:boto:Data: 2023-09-29T22:19:58.880 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.881 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.881 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.881 DEBUG:boto:Params: {} 2023-09-29T22:19:58.881 DEBUG:boto:Token: None 2023-09-29T22:19:58.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.881 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c4005feadafed7-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.884 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.885 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.885 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.885 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.885 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:58.885 DEBUG:boto:Data: 2023-09-29T22:19:58.885 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.885 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.886 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.886 DEBUG:boto:Params: {} 2023-09-29T22:19:58.886 DEBUG:boto:Token: None 2023-09-29T22:19:58.886 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy 2023-09-29T22:19:58.886 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0= 2023-09-29T22:19:58.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0=', 'Content-Length': '0'} 2023-09-29T22:19:58.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b4b1bfab3079324-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.889 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.889 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.889 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.889 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:58.889 DEBUG:boto:Data: 2023-09-29T22:19:58.890 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.890 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.890 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.890 DEBUG:boto:Params: {} 2023-09-29T22:19:58.890 DEBUG:boto:Token: None 2023-09-29T22:19:58.890 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy 2023-09-29T22:19:58.890 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0= 2023-09-29T22:19:58.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UA134SuPSSfKulODUQNlgvFTbz0=', 'Content-Length': '0'} 2023-09-29T22:19:58.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3a324ff291a9848-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:58.893 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.893 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.893 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.893 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.893 DEBUG:boto:Method: GET 2023-09-29T22:19:58.894 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.894 DEBUG:boto:Data: 2023-09-29T22:19:58.894 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.894 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.894 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.894 DEBUG:boto:Params: {} 2023-09-29T22:19:58.894 DEBUG:boto:Token: None 2023-09-29T22:19:58.894 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.895 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e13ae10c3c5719a-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.897 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:58.897 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:58.897 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.897 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.897 DEBUG:boto:Method: GET 2023-09-29T22:19:58.897 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.897 DEBUG:boto:Data: 2023-09-29T22:19:58.898 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.898 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.898 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.898 DEBUG:boto:Params: {} 2023-09-29T22:19:58.898 DEBUG:boto:Token: None 2023-09-29T22:19:58.898 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:58.898 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8= 2023-09-29T22:19:58.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b6m/bQMpBIEeyGb2W9A9HlxFLZ8=', 'Content-Length': '0'} 2023-09-29T22:19:58.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c474e7159c0e26b7-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.901 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.901 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.901 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.901 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.901 DEBUG:boto:Method: GET 2023-09-29T22:19:58.901 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.902 DEBUG:boto:Data: 2023-09-29T22:19:58.902 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.902 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.902 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.902 DEBUG:boto:Params: {} 2023-09-29T22:19:58.902 DEBUG:boto:Token: None 2023-09-29T22:19:58.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.903 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099330c72e822daed-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.906 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.906 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.906 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.907 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.907 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.907 DEBUG:boto:Method: GET 2023-09-29T22:19:58.907 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:58.907 DEBUG:boto:Data: 2023-09-29T22:19:58.907 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:58.907 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.907 DEBUG:boto:Params: {} 2023-09-29T22:19:58.907 DEBUG:boto:Token: None 2023-09-29T22:19:58.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw= 2023-09-29T22:19:58.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X66o32dPbwl5ZTqtD1ILvFsgZQw=', 'Content-Length': '0'} 2023-09-29T22:19:58.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9ccf9a77926ffc-0065174d8e-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.910 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.911 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone3, test-zone1} 2023-09-29T22:19:58.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone3, test-zone2} 2023-09-29T22:19:58.911 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.911 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.911 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.911 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.911 DEBUG:boto:Data: 2023-09-29T22:19:58.912 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.912 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.912 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.912 DEBUG:boto:Params: {} 2023-09-29T22:19:58.912 DEBUG:boto:Token: None 2023-09-29T22:19:58.912 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.913 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052eeabb69add61ea-0065174d8e-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.915 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.915 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.916 DEBUG:boto:Method: HEAD 2023-09-29T22:19:58.916 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:58.916 DEBUG:boto:Data: 2023-09-29T22:19:58.916 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.916 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.916 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.916 DEBUG:boto:Params: {} 2023-09-29T22:19:58.916 DEBUG:boto:Token: None 2023-09-29T22:19:58.916 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/ 2023-09-29T22:19:58.917 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0= 2023-09-29T22:19:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dca2Ld0y02Jvd2+ohhhL9gEOIk0=', 'Content-Length': '0'} 2023-09-29T22:19:58.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084e3b0326cfd19f3-0065174d8e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:19:58.919 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.919 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.919 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.919 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.919 DEBUG:boto:Method: GET 2023-09-29T22:19:58.919 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.919 DEBUG:boto:Data: 2023-09-29T22:19:58.920 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.920 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.920 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.920 DEBUG:boto:Params: {} 2023-09-29T22:19:58.920 DEBUG:boto:Token: None 2023-09-29T22:19:58.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d9945d7fd31cac-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.923 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.923 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.923 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:19:58.923 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.923 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.924 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.924 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.924 DEBUG:boto:Method: GET 2023-09-29T22:19:58.924 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.924 DEBUG:boto:Data: 2023-09-29T22:19:58.924 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.924 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.924 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.924 DEBUG:boto:Params: {} 2023-09-29T22:19:58.924 DEBUG:boto:Token: None 2023-09-29T22:19:58.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.925 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832c7695ab8b19ce-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.927 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.927 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:19:58.927 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.928 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.928 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.928 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.928 DEBUG:boto:Method: GET 2023-09-29T22:19:58.928 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.928 DEBUG:boto:Data: 2023-09-29T22:19:58.928 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.928 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:58.928 DEBUG:boto:Port: 8001 2023-09-29T22:19:58.928 DEBUG:boto:Params: {} 2023-09-29T22:19:58.929 DEBUG:boto:Token: None 2023-09-29T22:19:58.929 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.929 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee007385074c39f-0065174d8e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:58.974 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:58.975 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:58.975 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:58.975 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.975 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.975 DEBUG:boto:Method: GET 2023-09-29T22:19:58.976 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:19:58.976 DEBUG:boto:Data: 2023-09-29T22:19:58.976 DEBUG:boto:Headers: {} 2023-09-29T22:19:58.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:58.976 DEBUG:boto:Port: 8000 2023-09-29T22:19:58.976 DEBUG:boto:Params: {} 2023-09-29T22:19:58.976 DEBUG:boto:Token: None 2023-09-29T22:19:58.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:58 GMT /dmuhsu-19/?versions 2023-09-29T22:19:58.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0= 2023-09-29T22:19:58.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yLmtW7gKM48ENjAleT5cxklJnA0=', 'Content-Length': '0'} 2023-09-29T22:19:58.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a958a1714e60957-0065174d8e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.022 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:59.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:59.023 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.023 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.023 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.023 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.023 DEBUG:boto:Method: GET 2023-09-29T22:19:59.024 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.024 DEBUG:boto:Data: 2023-09-29T22:19:59.024 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.024 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.024 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.024 DEBUG:boto:Params: {} 2023-09-29T22:19:59.024 DEBUG:boto:Token: None 2023-09-29T22:19:59.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e16699e1a86a9905-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.027 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.027 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.027 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.027 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.027 DEBUG:boto:Method: GET 2023-09-29T22:19:59.028 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.028 DEBUG:boto:Data: 2023-09-29T22:19:59.028 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.028 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.028 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.028 DEBUG:boto:Params: {} 2023-09-29T22:19:59.028 DEBUG:boto:Token: None 2023-09-29T22:19:59.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000da8fd98d49ebbc14-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.030 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.031 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.031 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.031 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.031 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.031 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.031 DEBUG:boto:Data: 2023-09-29T22:19:59.031 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.032 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.032 DEBUG:boto:Params: {} 2023-09-29T22:19:59.032 DEBUG:boto:Token: None 2023-09-29T22:19:59.032 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.032 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aT+FGKHR2LDL6hc+atAvZw0Qxjs= 2023-09-29T22:19:59.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aT+FGKHR2LDL6hc+atAvZw0Qxjs=', 'Content-Length': '0'} 2023-09-29T22:19:59.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc094b9b3c638d94-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.034 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.034 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.035 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.035 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.035 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.035 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.035 DEBUG:boto:Data: 2023-09-29T22:19:59.035 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.035 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.035 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.036 DEBUG:boto:Params: {} 2023-09-29T22:19:59.036 DEBUG:boto:Token: None 2023-09-29T22:19:59.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.036 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aT+FGKHR2LDL6hc+atAvZw0Qxjs= 2023-09-29T22:19:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aT+FGKHR2LDL6hc+atAvZw0Qxjs=', 'Content-Length': '0'} 2023-09-29T22:19:59.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003547aaeb132e62e0-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:59.038 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.038 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.038 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.039 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.039 DEBUG:boto:Method: GET 2023-09-29T22:19:59.039 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.039 DEBUG:boto:Data: 2023-09-29T22:19:59.039 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.039 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.039 DEBUG:boto:Params: {} 2023-09-29T22:19:59.040 DEBUG:boto:Token: None 2023-09-29T22:19:59.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033742b60d1c7e51d-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.042 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.042 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.042 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.042 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.043 DEBUG:boto:Method: GET 2023-09-29T22:19:59.043 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.043 DEBUG:boto:Data: 2023-09-29T22:19:59.043 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.043 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.043 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.043 DEBUG:boto:Params: {} 2023-09-29T22:19:59.043 DEBUG:boto:Token: None 2023-09-29T22:19:59.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009982b64e43154732-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.046 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.046 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.046 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.046 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.046 DEBUG:boto:Method: GET 2023-09-29T22:19:59.046 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.046 DEBUG:boto:Data: 2023-09-29T22:19:59.046 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.046 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.047 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.047 DEBUG:boto:Params: {} 2023-09-29T22:19:59.047 DEBUG:boto:Token: None 2023-09-29T22:19:59.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8= 2023-09-29T22:19:59.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8=', 'Content-Length': '0'} 2023-09-29T22:19:59.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2885c7502bdf0d9-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.050 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:59.050 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.050 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.050 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.050 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.051 DEBUG:boto:Method: GET 2023-09-29T22:19:59.051 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.051 DEBUG:boto:Data: 2023-09-29T22:19:59.051 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.051 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.051 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.051 DEBUG:boto:Params: {} 2023-09-29T22:19:59.051 DEBUG:boto:Token: None 2023-09-29T22:19:59.051 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8= 2023-09-29T22:19:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8=', 'Content-Length': '0'} 2023-09-29T22:19:59.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeb1c3a3f6834061-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.054 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:59.054 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.054 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.054 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.054 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:59.055 DEBUG:boto:Data: 2023-09-29T22:19:59.055 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.055 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.055 DEBUG:boto:Params: {} 2023-09-29T22:19:59.055 DEBUG:boto:Token: None 2023-09-29T22:19:59.055 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy 2023-09-29T22:19:59.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XM28SB3XPb/Psfw9ekD8tFUXzlc= 2023-09-29T22:19:59.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XM28SB3XPb/Psfw9ekD8tFUXzlc=', 'Content-Length': '0'} 2023-09-29T22:19:59.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6913d5a9864dab5-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.057 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.058 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.058 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.058 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:19:59.058 DEBUG:boto:Data: 2023-09-29T22:19:59.058 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.058 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.058 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.058 DEBUG:boto:Params: {} 2023-09-29T22:19:59.058 DEBUG:boto:Token: None 2023-09-29T22:19:59.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy 2023-09-29T22:19:59.059 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XM28SB3XPb/Psfw9ekD8tFUXzlc= 2023-09-29T22:19:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XM28SB3XPb/Psfw9ekD8tFUXzlc=', 'Content-Length': '0'} 2023-09-29T22:19:59.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077633aabc9b6bc6c-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:19:59.064 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.064 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.064 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.065 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.065 DEBUG:boto:Method: GET 2023-09-29T22:19:59.065 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.065 DEBUG:boto:Data: 2023-09-29T22:19:59.065 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.065 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.065 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.065 DEBUG:boto:Params: {} 2023-09-29T22:19:59.065 DEBUG:boto:Token: None 2023-09-29T22:19:59.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.066 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062fb366e2979b391-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.068 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:19:59.068 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:19:59.068 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.068 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.068 DEBUG:boto:Method: GET 2023-09-29T22:19:59.068 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.068 DEBUG:boto:Data: 2023-09-29T22:19:59.068 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.068 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.068 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.069 DEBUG:boto:Params: {} 2023-09-29T22:19:59.069 DEBUG:boto:Token: None 2023-09-29T22:19:59.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:19:59.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw= 2023-09-29T22:19:59.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j/RjNyFXHOxuVc29QEYvlNIAbAw=', 'Content-Length': '0'} 2023-09-29T22:19:59.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b17d54d670a9dee-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.071 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.071 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.071 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.071 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.071 DEBUG:boto:Method: GET 2023-09-29T22:19:59.071 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.071 DEBUG:boto:Data: 2023-09-29T22:19:59.071 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.072 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.072 DEBUG:boto:Params: {} 2023-09-29T22:19:59.072 DEBUG:boto:Token: None 2023-09-29T22:19:59.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8= 2023-09-29T22:19:59.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8=', 'Content-Length': '0'} 2023-09-29T22:19:59.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9f7e93a1e16967-0065174d8f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.074 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:59.075 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.075 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.075 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.075 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.075 DEBUG:boto:Method: GET 2023-09-29T22:19:59.075 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:19:59.075 DEBUG:boto:Data: 2023-09-29T22:19:59.075 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.075 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.076 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.076 DEBUG:boto:Params: {} 2023-09-29T22:19:59.076 DEBUG:boto:Token: None 2023-09-29T22:19:59.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8= 2023-09-29T22:19:59.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cFO65ESLEioAVoFwLwtVLi4sWw8=', 'Content-Length': '0'} 2023-09-29T22:19:59.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862a3ab464416930-0065174d8f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.122 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:19:59.123 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone3, test-zone2} 2023-09-29T22:19:59.123 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.123 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.123 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.123 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.123 DEBUG:boto:Data: 2023-09-29T22:19:59.124 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.124 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.124 DEBUG:boto:Params: {} 2023-09-29T22:19:59.124 DEBUG:boto:Token: None 2023-09-29T22:19:59.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d30771bbcf82edf-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.127 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.127 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.127 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.127 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.127 DEBUG:boto:Data: 2023-09-29T22:19:59.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.128 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.128 DEBUG:boto:Params: {} 2023-09-29T22:19:59.128 DEBUG:boto:Token: None 2023-09-29T22:19:59.128 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-amz-request-id', 'tx00000a370a327e20f14c0-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.138 DEBUG:rgw_multi.tests:zone=test-zone1 version=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:19:59.139 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.139 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.139 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.139 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.139 DEBUG:boto:Data: 2023-09-29T22:19:59.139 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.139 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.139 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.139 DEBUG:boto:Params: {} 2023-09-29T22:19:59.140 DEBUG:boto:Token: None 2023-09-29T22:19:59.140 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.140 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019055f207b3b7c35-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.142 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.143 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.143 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.143 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.143 DEBUG:boto:Data: 2023-09-29T22:19:59.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.143 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.143 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.143 DEBUG:boto:Params: {} 2023-09-29T22:19:59.143 DEBUG:boto:Token: None 2023-09-29T22:19:59.144 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-amz-request-id', 'tx00000d18bd97ba2ba77a0-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.154 DEBUG:rgw_multi.tests:zone=test-zone2 version=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:19:59.154 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.154 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.154 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.154 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.154 DEBUG:boto:Data: 2023-09-29T22:19:59.155 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.155 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.155 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.155 DEBUG:boto:Params: {} 2023-09-29T22:19:59.155 DEBUG:boto:Token: None 2023-09-29T22:19:59.155 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d287f9b67019fab1-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.158 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.158 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.158 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.158 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.158 DEBUG:boto:Data: 2023-09-29T22:19:59.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.158 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.159 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.159 DEBUG:boto:Params: {} 2023-09-29T22:19:59.159 DEBUG:boto:Token: None 2023-09-29T22:19:59.159 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.159 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-amz-request-id', 'tx0000013a3db8186b10dba-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.171 DEBUG:rgw_multi.tests:zone=test-zone3 version=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:19:59.171 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.171 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.171 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.171 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.171 DEBUG:boto:Data: 2023-09-29T22:19:59.172 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.172 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.172 DEBUG:boto:Params: {} 2023-09-29T22:19:59.172 DEBUG:boto:Token: None 2023-09-29T22:19:59.172 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d66ec412aa5dd2ef-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.175 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.175 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.175 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.175 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.175 DEBUG:boto:Data: 2023-09-29T22:19:59.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.175 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.175 DEBUG:boto:Params: {} 2023-09-29T22:19:59.176 DEBUG:boto:Token: None 2023-09-29T22:19:59.176 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-amz-request-id', 'tx00000f7cc2a601ab7300f-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.219 DEBUG:rgw_multi.tests:zone=test-zone1 version=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:19:59.219 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.220 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.220 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.220 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.220 DEBUG:boto:Data: 2023-09-29T22:19:59.220 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.220 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.221 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.221 DEBUG:boto:Params: {} 2023-09-29T22:19:59.221 DEBUG:boto:Token: None 2023-09-29T22:19:59.221 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.222 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000751009710b794dfb-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.224 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.224 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.224 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.225 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.225 DEBUG:boto:Data: 2023-09-29T22:19:59.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.225 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.225 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.225 DEBUG:boto:Params: {} 2023-09-29T22:19:59.225 DEBUG:boto:Token: None 2023-09-29T22:19:59.225 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-amz-request-id', 'tx000000eb3f3c7d5c4208b-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.234 DEBUG:rgw_multi.tests:zone=test-zone2 version=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:19:59.234 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.234 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.234 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.234 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.235 DEBUG:boto:Data: 2023-09-29T22:19:59.235 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.235 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.235 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.235 DEBUG:boto:Params: {} 2023-09-29T22:19:59.235 DEBUG:boto:Token: None 2023-09-29T22:19:59.235 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.235 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042d9f23459f1ea32-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.238 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.238 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.238 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.238 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.238 DEBUG:boto:Data: 2023-09-29T22:19:59.238 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.239 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.239 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.239 DEBUG:boto:Params: {} 2023-09-29T22:19:59.239 DEBUG:boto:Token: None 2023-09-29T22:19:59.239 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.239 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-amz-request-id', 'tx000005527acf2eaf9b52f-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.248 DEBUG:rgw_multi.tests:zone=test-zone3 version=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:19:59.248 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.248 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.248 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.248 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.248 DEBUG:boto:Data: 2023-09-29T22:19:59.249 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.249 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.249 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.249 DEBUG:boto:Params: {} 2023-09-29T22:19:59.249 DEBUG:boto:Token: None 2023-09-29T22:19:59.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.249 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa9e8810f176c41d-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.252 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.252 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.252 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.252 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.252 DEBUG:boto:Data: 2023-09-29T22:19:59.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.252 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.252 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.253 DEBUG:boto:Params: {} 2023-09-29T22:19:59.253 DEBUG:boto:Token: None 2023-09-29T22:19:59.253 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.253 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-amz-request-id', 'tx000006cfcbc44d56772fb-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.262 DEBUG:rgw_multi.tests:zone=test-zone1 version=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:19:59.262 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.262 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.262 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.262 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.262 DEBUG:boto:Data: 2023-09-29T22:19:59.262 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.262 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.262 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.263 DEBUG:boto:Params: {} 2023-09-29T22:19:59.263 DEBUG:boto:Token: None 2023-09-29T22:19:59.263 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.263 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d1a954864149624-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.265 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.266 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.266 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.266 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.266 DEBUG:boto:Data: 2023-09-29T22:19:59.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.266 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.266 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.266 DEBUG:boto:Params: {} 2023-09-29T22:19:59.266 DEBUG:boto:Token: None 2023-09-29T22:19:59.267 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-amz-request-id', 'tx00000e4d1f7f816995288-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.277 DEBUG:rgw_multi.tests:zone=test-zone2 version=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:19:59.277 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.277 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.277 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.277 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.277 DEBUG:boto:Data: 2023-09-29T22:19:59.277 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.277 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.277 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.278 DEBUG:boto:Params: {} 2023-09-29T22:19:59.278 DEBUG:boto:Token: None 2023-09-29T22:19:59.278 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.278 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000597735805cd79b02-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.280 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.281 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.281 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.281 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.281 DEBUG:boto:Data: 2023-09-29T22:19:59.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.281 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.281 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.281 DEBUG:boto:Params: {} 2023-09-29T22:19:59.282 DEBUG:boto:Token: None 2023-09-29T22:19:59.282 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.282 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-amz-request-id', 'tx0000084e7fe0ba211a5b1-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.327 DEBUG:rgw_multi.tests:zone=test-zone3 version=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:19:59.327 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.328 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.328 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.328 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.328 DEBUG:boto:Data: 2023-09-29T22:19:59.328 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.328 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.329 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.329 DEBUG:boto:Params: {} 2023-09-29T22:19:59.329 DEBUG:boto:Token: None 2023-09-29T22:19:59.329 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.330 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a238a253df725a38-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.332 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.333 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.333 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.333 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.333 DEBUG:boto:Data: 2023-09-29T22:19:59.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.333 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.334 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.334 DEBUG:boto:Params: {} 2023-09-29T22:19:59.334 DEBUG:boto:Token: None 2023-09-29T22:19:59.334 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.335 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-amz-request-id', 'tx000004cab552ed61f01d3-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.344 DEBUG:rgw_multi.tests:zone=test-zone1 version=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:19:59.344 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.345 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.345 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.345 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.345 DEBUG:boto:Data: 2023-09-29T22:19:59.346 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.346 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.346 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.346 DEBUG:boto:Params: {} 2023-09-29T22:19:59.346 DEBUG:boto:Token: None 2023-09-29T22:19:59.347 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccc8b748e7d2b8a5-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.350 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.350 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.350 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.350 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.351 DEBUG:boto:Data: 2023-09-29T22:19:59.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.351 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.351 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.351 DEBUG:boto:Params: {} 2023-09-29T22:19:59.351 DEBUG:boto:Token: None 2023-09-29T22:19:59.352 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.352 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-amz-request-id', 'tx00000cea9b73795d3c103-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.362 DEBUG:rgw_multi.tests:zone=test-zone2 version=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:19:59.362 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.362 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.362 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.362 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.362 DEBUG:boto:Data: 2023-09-29T22:19:59.362 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.362 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.363 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.363 DEBUG:boto:Params: {} 2023-09-29T22:19:59.363 DEBUG:boto:Token: None 2023-09-29T22:19:59.363 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.363 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013d3d335a04fc3f3-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.365 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.366 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.366 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.366 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.366 DEBUG:boto:Data: 2023-09-29T22:19:59.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.366 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.366 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.366 DEBUG:boto:Params: {} 2023-09-29T22:19:59.366 DEBUG:boto:Token: None 2023-09-29T22:19:59.367 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.367 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-amz-request-id', 'tx00000aeb34f9dc1955c69-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.377 DEBUG:rgw_multi.tests:zone=test-zone3 version=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:19:59.378 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.378 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.378 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.378 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.378 DEBUG:boto:Data: 2023-09-29T22:19:59.378 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.378 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.378 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.379 DEBUG:boto:Params: {} 2023-09-29T22:19:59.379 DEBUG:boto:Token: None 2023-09-29T22:19:59.379 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.379 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e374ebe0949a5ce-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.381 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.382 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.382 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.382 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.382 DEBUG:boto:Data: 2023-09-29T22:19:59.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.382 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.382 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.382 DEBUG:boto:Params: {} 2023-09-29T22:19:59.383 DEBUG:boto:Token: None 2023-09-29T22:19:59.383 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-amz-request-id', 'tx00000f4a1a7b3ef8f6318-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.427 DEBUG:rgw_multi.tests:zone=test-zone1 version=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:19:59.427 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.427 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.427 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.428 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.428 DEBUG:boto:Data: 2023-09-29T22:19:59.428 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.428 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.428 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.428 DEBUG:boto:Params: {} 2023-09-29T22:19:59.429 DEBUG:boto:Token: None 2023-09-29T22:19:59.429 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.429 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025e61a48b3f75ca0-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.432 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.432 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.433 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.433 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.433 DEBUG:boto:Data: 2023-09-29T22:19:59.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.433 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.433 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.434 DEBUG:boto:Params: {} 2023-09-29T22:19:59.434 DEBUG:boto:Token: None 2023-09-29T22:19:59.434 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.434 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-amz-request-id', 'tx000005aaf5c47f2eae86d-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.479 DEBUG:rgw_multi.tests:zone=test-zone2 version=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:19:59.479 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.479 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.479 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.479 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.479 DEBUG:boto:Data: 2023-09-29T22:19:59.479 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.480 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.480 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.480 DEBUG:boto:Params: {} 2023-09-29T22:19:59.480 DEBUG:boto:Token: None 2023-09-29T22:19:59.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b5df4259d19381-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.482 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.483 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.483 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.483 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.483 DEBUG:boto:Data: 2023-09-29T22:19:59.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.483 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.483 DEBUG:boto:Params: {} 2023-09-29T22:19:59.483 DEBUG:boto:Token: None 2023-09-29T22:19:59.484 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-amz-request-id', 'tx00000cf3805bc91248307-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.492 DEBUG:rgw_multi.tests:zone=test-zone3 version=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:19:59.492 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.492 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.492 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.492 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.492 DEBUG:boto:Data: 2023-09-29T22:19:59.493 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.493 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.493 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.493 DEBUG:boto:Params: {} 2023-09-29T22:19:59.493 DEBUG:boto:Token: None 2023-09-29T22:19:59.493 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.493 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f415a8aaa395cd49-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.495 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.496 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.496 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.496 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.496 DEBUG:boto:Data: 2023-09-29T22:19:59.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.496 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.496 DEBUG:boto:Params: {} 2023-09-29T22:19:59.497 DEBUG:boto:Token: None 2023-09-29T22:19:59.497 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.497 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-amz-request-id', 'tx0000045d4cab813400f69-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.505 DEBUG:rgw_multi.tests:zone=test-zone1 version=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:19:59.505 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.505 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.505 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.505 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.505 DEBUG:boto:Data: 2023-09-29T22:19:59.505 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.505 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.505 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.506 DEBUG:boto:Params: {} 2023-09-29T22:19:59.506 DEBUG:boto:Token: None 2023-09-29T22:19:59.506 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.506 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088c14ac37123fed4-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.509 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.509 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.509 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.509 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.509 DEBUG:boto:Data: 2023-09-29T22:19:59.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.509 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.509 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.510 DEBUG:boto:Params: {} 2023-09-29T22:19:59.510 DEBUG:boto:Token: None 2023-09-29T22:19:59.510 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.510 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-amz-request-id', 'tx00000675d1d5406ee288c-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.518 DEBUG:rgw_multi.tests:zone=test-zone2 version=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:19:59.519 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.519 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.519 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.519 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.519 DEBUG:boto:Data: 2023-09-29T22:19:59.519 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.519 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.519 DEBUG:boto:Params: {} 2023-09-29T22:19:59.520 DEBUG:boto:Token: None 2023-09-29T22:19:59.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.520 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d78e22400aa7c58-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.522 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.523 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.523 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.523 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.523 DEBUG:boto:Data: 2023-09-29T22:19:59.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.523 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.523 DEBUG:boto:Params: {} 2023-09-29T22:19:59.524 DEBUG:boto:Token: None 2023-09-29T22:19:59.524 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-amz-request-id', 'tx00000120d01995f945709-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.567 DEBUG:rgw_multi.tests:zone=test-zone3 version=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:19:59.567 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.567 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.567 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.567 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.568 DEBUG:boto:Data: 2023-09-29T22:19:59.568 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.568 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.568 DEBUG:boto:Params: {} 2023-09-29T22:19:59.568 DEBUG:boto:Token: None 2023-09-29T22:19:59.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae106c40b8270be0-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.571 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.571 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.571 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.571 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.571 DEBUG:boto:Data: 2023-09-29T22:19:59.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.572 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.572 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.572 DEBUG:boto:Params: {} 2023-09-29T22:19:59.572 DEBUG:boto:Token: None 2023-09-29T22:19:59.572 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.572 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-amz-request-id', 'tx00000eef61f5b8534aa32-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.583 DEBUG:rgw_multi.tests:zone=test-zone1 version=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:19:59.583 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.583 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.584 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.584 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.584 DEBUG:boto:Data: 2023-09-29T22:19:59.584 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.584 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.584 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.584 DEBUG:boto:Params: {} 2023-09-29T22:19:59.584 DEBUG:boto:Token: None 2023-09-29T22:19:59.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.585 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d7079157081019-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.587 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.587 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.587 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.588 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.588 DEBUG:boto:Data: 2023-09-29T22:19:59.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.588 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.588 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.588 DEBUG:boto:Params: {} 2023-09-29T22:19:59.588 DEBUG:boto:Token: None 2023-09-29T22:19:59.588 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-amz-request-id', 'tx000001017e8aae2f91496-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.599 DEBUG:rgw_multi.tests:zone=test-zone2 version=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:19:59.599 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.599 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.599 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.599 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.599 DEBUG:boto:Data: 2023-09-29T22:19:59.599 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.599 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.600 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.600 DEBUG:boto:Params: {} 2023-09-29T22:19:59.600 DEBUG:boto:Token: None 2023-09-29T22:19:59.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.600 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002922a4da9e602426-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.603 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.603 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.603 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.603 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.603 DEBUG:boto:Data: 2023-09-29T22:19:59.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.603 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.603 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.604 DEBUG:boto:Params: {} 2023-09-29T22:19:59.604 DEBUG:boto:Token: None 2023-09-29T22:19:59.604 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-amz-request-id', 'tx000009385f8ee331e634e-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.613 DEBUG:rgw_multi.tests:zone=test-zone3 version=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:19:59.614 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.614 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.614 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.614 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.614 DEBUG:boto:Data: 2023-09-29T22:19:59.614 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.614 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.614 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.614 DEBUG:boto:Params: {} 2023-09-29T22:19:59.615 DEBUG:boto:Token: None 2023-09-29T22:19:59.615 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.615 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea32525b72b4634f-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.617 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.617 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.617 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.618 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.618 DEBUG:boto:Data: 2023-09-29T22:19:59.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.618 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.618 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.618 DEBUG:boto:Params: {} 2023-09-29T22:19:59.618 DEBUG:boto:Token: None 2023-09-29T22:19:59.618 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.619 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-amz-request-id', 'tx00000889bf3204a7ffeea-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.663 DEBUG:rgw_multi.tests:zone=test-zone1 version=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:19:59.663 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.663 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.663 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.663 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.663 DEBUG:boto:Data: 2023-09-29T22:19:59.664 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.664 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.664 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.664 DEBUG:boto:Params: {} 2023-09-29T22:19:59.664 DEBUG:boto:Token: None 2023-09-29T22:19:59.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007da6ac9f9c858f61-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.667 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.667 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.667 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.667 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.667 DEBUG:boto:Data: 2023-09-29T22:19:59.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.668 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.668 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.668 DEBUG:boto:Params: {} 2023-09-29T22:19:59.668 DEBUG:boto:Token: None 2023-09-29T22:19:59.668 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-amz-request-id', 'tx000004c873774d0783bea-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.711 DEBUG:rgw_multi.tests:zone=test-zone2 version=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:19:59.711 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.711 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.711 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.712 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.712 DEBUG:boto:Data: 2023-09-29T22:19:59.712 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.712 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.712 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.712 DEBUG:boto:Params: {} 2023-09-29T22:19:59.712 DEBUG:boto:Token: None 2023-09-29T22:19:59.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d98c00ad86edcd7-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.715 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.715 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.715 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.715 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.715 DEBUG:boto:Data: 2023-09-29T22:19:59.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.716 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.716 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.716 DEBUG:boto:Params: {} 2023-09-29T22:19:59.716 DEBUG:boto:Token: None 2023-09-29T22:19:59.716 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-amz-request-id', 'tx0000098191e7c51deed0b-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.759 DEBUG:rgw_multi.tests:zone=test-zone3 version=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:19:59.759 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.759 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.759 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.759 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.759 DEBUG:boto:Data: 2023-09-29T22:19:59.760 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.760 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.760 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.760 DEBUG:boto:Params: {} 2023-09-29T22:19:59.760 DEBUG:boto:Token: None 2023-09-29T22:19:59.760 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6aed391c4d0ad5f-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.763 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.763 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.763 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.763 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.763 DEBUG:boto:Data: 2023-09-29T22:19:59.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.763 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.763 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.763 DEBUG:boto:Params: {} 2023-09-29T22:19:59.764 DEBUG:boto:Token: None 2023-09-29T22:19:59.764 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.764 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-amz-request-id', 'tx000006c00352753444af1-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.772 DEBUG:rgw_multi.tests:zone=test-zone1 version=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:19:59.772 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.772 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.772 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.772 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.772 DEBUG:boto:Data: 2023-09-29T22:19:59.772 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.772 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.772 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.772 DEBUG:boto:Params: {} 2023-09-29T22:19:59.773 DEBUG:boto:Token: None 2023-09-29T22:19:59.773 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.773 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029ed4e6e8dbbe5be-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.775 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.775 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.775 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.775 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.776 DEBUG:boto:Data: 2023-09-29T22:19:59.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.776 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.776 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.776 DEBUG:boto:Params: {} 2023-09-29T22:19:59.776 DEBUG:boto:Token: None 2023-09-29T22:19:59.776 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.776 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-amz-request-id', 'tx00000f543fe9ab0f4b785-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.786 DEBUG:rgw_multi.tests:zone=test-zone2 version=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:19:59.787 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.787 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.787 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.787 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.787 DEBUG:boto:Data: 2023-09-29T22:19:59.787 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.787 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.787 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.788 DEBUG:boto:Params: {} 2023-09-29T22:19:59.788 DEBUG:boto:Token: None 2023-09-29T22:19:59.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d255553278d72d-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.790 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.791 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.791 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.791 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.791 DEBUG:boto:Data: 2023-09-29T22:19:59.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.791 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.791 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.791 DEBUG:boto:Params: {} 2023-09-29T22:19:59.792 DEBUG:boto:Token: None 2023-09-29T22:19:59.792 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-amz-request-id', 'tx0000087d33eb7dc2a5e67-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.800 DEBUG:rgw_multi.tests:zone=test-zone3 version=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:19:59.800 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.800 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.800 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.800 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.800 DEBUG:boto:Data: 2023-09-29T22:19:59.800 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.801 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.801 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.801 DEBUG:boto:Params: {} 2023-09-29T22:19:59.801 DEBUG:boto:Token: None 2023-09-29T22:19:59.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.801 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f86b4220b2028a6-0065174d8f-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.803 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.804 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.804 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.804 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.804 DEBUG:boto:Data: 2023-09-29T22:19:59.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:19:59.804 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.804 DEBUG:boto:Params: {} 2023-09-29T22:19:59.805 DEBUG:boto:Token: None 2023-09-29T22:19:59.805 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-amz-request-id', 'tx00000a2651673abb84ba6-0065174d8f-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.851 DEBUG:rgw_multi.tests:zone=test-zone1 version=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:19:59.851 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.851 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.851 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.851 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.851 DEBUG:boto:Data: 2023-09-29T22:19:59.852 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.852 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.852 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.852 DEBUG:boto:Params: {} 2023-09-29T22:19:59.852 DEBUG:boto:Token: None 2023-09-29T22:19:59.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000906d0fb78e93d13a-0065174d8f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.855 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.855 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.855 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.855 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.855 DEBUG:boto:Data: 2023-09-29T22:19:59.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.855 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:19:59.855 DEBUG:boto:Port: 8000 2023-09-29T22:19:59.855 DEBUG:boto:Params: {} 2023-09-29T22:19:59.856 DEBUG:boto:Token: None 2023-09-29T22:19:59.856 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-amz-request-id', 'tx000003a6b1b2f31dfe973-0065174d8f-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.865 DEBUG:rgw_multi.tests:zone=test-zone2 version=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:19:59.865 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:19:59.866 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:19:59.866 DEBUG:boto:Method: HEAD 2023-09-29T22:19:59.866 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:19:59.866 DEBUG:boto:Data: 2023-09-29T22:19:59.866 DEBUG:boto:Headers: {} 2023-09-29T22:19:59.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.866 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.866 DEBUG:boto:Params: {} 2023-09-29T22:19:59.867 DEBUG:boto:Token: None 2023-09-29T22:19:59.867 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/ 2023-09-29T22:19:59.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s= 2023-09-29T22:19:59.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3kAU6Ga9NHjRFXDa5y9CoRvXU8s=', 'Content-Length': '0'} 2023-09-29T22:19:59.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3b7b232734d2639-0065174d8f-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.869 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:19:59.869 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:19:59.870 DEBUG:boto:Method: PUT 2023-09-29T22:19:59.870 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:19:59.870 DEBUG:boto:Data: 2023-09-29T22:19:59.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:19:59.870 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:19:59.870 DEBUG:boto:Port: 8001 2023-09-29T22:19:59.870 DEBUG:boto:Params: {} 2023-09-29T22:19:59.870 DEBUG:boto:Token: None 2023-09-29T22:19:59.870 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:19:59 GMT /dmuhsu-19/obj 2023-09-29T22:19:59.871 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk= 2023-09-29T22:19:59.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:19:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iRb05jbrjeWTksi+/SWRs2JKsvk='} 2023-09-29T22:19:59.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-amz-request-id', 'tx00000a17925b32b52318f-0065174d8f-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:19:59.878 DEBUG:rgw_multi.tests:zone=test-zone3 version=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:19:59.878 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-19 2023-09-29T22:19:59.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:59.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:19:59.879 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:19:59.932 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:19:59.932 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:19:59.966 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:19:59.966+0000 7ff5cbc17880 1 waiting for incremental sync to catch up: 2023-09-29T22:19:59.966 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:19:59.966 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:04.967 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:20:04.968+0000 7ff5cbc17880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:04.968 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:20:04.968 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:09.969 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:20:09.969+0000 7ff5cbc17880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:09.969 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:20:09.969 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:14.970 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:20:14.970+0000 7ff5cbc17880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:14.970 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.24.10, , , , , ] 2023-09-29T22:20:14.970 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:19.971 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:20:19.971+0000 7ff5cbc17880 1 bucket sync caught up with source: 2023-09-29T22:20:19.972 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:19.972 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:19.972 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:20:19.971+0000 7ff5cbc17880 0 bucket checkpoint complete 2023-09-29T22:20:19.987 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-19 2023-09-29T22:20:19.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:20:19.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:20:19.988 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:20:20.047 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:20:20.047 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:20:20.084 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:20.084+0000 7eff7c1dc880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:20.084 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.26.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:20.084 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000005.28.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:25.085+0000 7eff7c1dc880 1 bucket sync caught up with source: 2023-09-29T22:20:25.085 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.28.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:25.085 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000005.28.10, , , 00000000020.46.12, , ] 2023-09-29T22:20:25.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:25.085+0000 7eff7c1dc880 0 bucket checkpoint complete 2023-09-29T22:20:25.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-19 2023-09-29T22:20:25.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:20:25.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:20:25.100 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:20:25.155 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:20:25.155 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:20:25.192 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:25.192+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:25.192 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000001.23.6, , ] 2023-09-29T22:20:25.192 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:30.193 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:30.193+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:30.194 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000001.23.6, , ] 2023-09-29T22:20:30.194 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:35.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:35.194+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:35.194 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000001.23.6, , ] 2023-09-29T22:20:35.195 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:40.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:40.195+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:40.196 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:20:40.196 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:45.197 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:45.196+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:45.197 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:20:45.197 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:50.198 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:50.197+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:50.198 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:20:50.198 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:20:55.199 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:20:55.198+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:20:55.199 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:20:55.199 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:00.200 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:00.200+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:00.201 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:21:00.201 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:05.201 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:05.201+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:05.201 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:21:05.202 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:10.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:10.202+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:10.203 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:21:10.203 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:15.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:15.204+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:15.204 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000005.31.6, , ] 2023-09-29T22:21:15.204 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:20.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:20.205+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:20.205 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:20.206 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:25.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:25.206+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:25.206 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:25.206 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:30.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:30.207+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:30.207 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:30.207 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:35.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:35.208+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:35.208 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:35.208 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:40.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:40.209+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:40.210 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:40.210 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:45.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:45.210+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:45.211 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:45.211 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:50.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:50.211+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:50.212 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:50.212 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:21:55.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:21:55.213+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:21:55.214 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:21:55.214 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:00.215+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:00.215 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:00.215 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:05.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:05.216+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:05.216 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:05.216 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:10.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:10.217+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:10.217 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:10.217 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:15.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:15.218+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:15.219 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:15.219 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:20.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:20.220+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:20.220 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:20.220 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:25.221+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:25.221 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:25.221 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:30.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:30.222+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:30.222 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:30.222 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:35.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:35.223+0000 7f8c87729880 1 waiting for incremental sync to catch up: 2023-09-29T22:22:35.224 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000028.84.6, , ] 2023-09-29T22:22:35.224 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:37.504 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:22:37.503+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:22:40.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.223+0000 7f8c87729880 1 bucket sync caught up with source: 2023-09-29T22:22:40.225 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000044.126.10, , ] 2023-09-29T22:22:40.225 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000043.124.10, , ] 2023-09-29T22:22:40.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.223+0000 7f8c87729880 0 bucket checkpoint complete 2023-09-29T22:22:40.240 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-19 2023-09-29T22:22:40.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.241 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:22:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:40.298 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:40.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.334+0000 7f0b67d30880 1 bucket sync caught up with source: 2023-09-29T22:22:40.336 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.47.10, , , 00000000044.126.10, , ] 2023-09-29T22:22:40.336 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.47.10, , , 00000000044.126.10, , ] 2023-09-29T22:22:40.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.334+0000 7f0b67d30880 0 bucket checkpoint complete 2023-09-29T22:22:40.348 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-19 2023-09-29T22:22:40.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.349 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:22:40.401 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:40.402 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.435+0000 7f4da3411880 1 bucket sync caught up with source: 2023-09-29T22:22:40.437 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000044.127.10, , ] 2023-09-29T22:22:40.437 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.127.10, , ] 2023-09-29T22:22:40.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:22:40.436+0000 7f4da3411880 0 bucket checkpoint complete 2023-09-29T22:22:40.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-19 2023-09-29T22:22:40.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:22:40.449 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:22:40.509 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:40.509 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:40.548 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:22:40.548+0000 7fa110015880 1 bucket sync caught up with source: 2023-09-29T22:22:40.548 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.59.10, , , 00000000044.127.10, , ] 2023-09-29T22:22:40.548 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000003.59.10, , , 00000000044.127.10, , ] 2023-09-29T22:22:40.548 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:22:40.548+0000 7fa110015880 0 bucket checkpoint complete 2023-09-29T22:22:40.561 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone2, test-zone1} 2023-09-29T22:22:40.561 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.561 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.561 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.561 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:40.561 DEBUG:boto:Data: 2023-09-29T22:22:40.562 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.562 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.562 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.562 DEBUG:boto:Params: {} 2023-09-29T22:22:40.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:22:40.562 DEBUG:boto:Token: None 2023-09-29T22:22:40.563 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.563 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vzGl7iXDYqXrZziGqU96idAExpw= 2023-09-29T22:22:40.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vzGl7iXDYqXrZziGqU96idAExpw=', 'Content-Length': '0'} 2023-09-29T22:22:40.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b07f2c9d989a4041-0065174e30-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.567 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.567 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.567 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.567 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:40.568 DEBUG:boto:Data: 2023-09-29T22:22:40.568 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.568 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.568 DEBUG:boto:Params: {} 2023-09-29T22:22:40.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:22:40.568 DEBUG:boto:Token: None 2023-09-29T22:22:40.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vzGl7iXDYqXrZziGqU96idAExpw= 2023-09-29T22:22:40.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vzGl7iXDYqXrZziGqU96idAExpw=', 'Content-Length': '0'} 2023-09-29T22:22:40.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008038216849542f1f-0065174e30-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.572 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:22:40.572 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.572 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.572 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.572 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.572 DEBUG:boto:Method: GET 2023-09-29T22:22:40.573 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.573 DEBUG:boto:Data: 2023-09-29T22:22:40.573 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.573 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.573 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.573 DEBUG:boto:Params: {} 2023-09-29T22:22:40.573 DEBUG:boto:Token: None 2023-09-29T22:22:40.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/?versions 2023-09-29T22:22:40.573 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk= 2023-09-29T22:22:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk=', 'Content-Length': '0'} 2023-09-29T22:22:40.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129db4a195052fed-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.577 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.578 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:22:40.582 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.582 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.582 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.582 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.582 DEBUG:boto:Method: GET 2023-09-29T22:22:40.582 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.582 DEBUG:boto:Data: 2023-09-29T22:22:40.582 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.582 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.582 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.582 DEBUG:boto:Params: {} 2023-09-29T22:22:40.583 DEBUG:boto:Token: None 2023-09-29T22:22:40.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/?versions 2023-09-29T22:22:40.583 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk= 2023-09-29T22:22:40.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk=', 'Content-Length': '0'} 2023-09-29T22:22:40.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad0b008ea68d0f0-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.586 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.587 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:40.590 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.590 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.591 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.591 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.591 DEBUG:boto:Method: GET 2023-09-29T22:22:40.591 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.591 DEBUG:boto:Data: 2023-09-29T22:22:40.591 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.591 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.591 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.591 DEBUG:boto:Params: {} 2023-09-29T22:22:40.591 DEBUG:boto:Token: None 2023-09-29T22:22:40.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/?versions 2023-09-29T22:22:40.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk= 2023-09-29T22:22:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk=', 'Content-Length': '0'} 2023-09-29T22:22:40.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a89bf8f28d7cc7bc-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.594 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.596 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.596 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.596 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.596 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.596 DEBUG:boto:Method: GET 2023-09-29T22:22:40.597 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:40.597 DEBUG:boto:Data: 2023-09-29T22:22:40.597 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.597 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.597 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.597 DEBUG:boto:Params: {} 2023-09-29T22:22:40.597 DEBUG:boto:Token: None 2023-09-29T22:22:40.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/?versions 2023-09-29T22:22:40.597 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk= 2023-09-29T22:22:40.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yCZMlsvYvUvMo3mPJmX1PB/s7xk=', 'Content-Length': '0'} 2023-09-29T22:22:40.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e088fe7001d094d3-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.600 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:40.602 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.602 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.602 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.602 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.602 DEBUG:boto:Method: GET 2023-09-29T22:22:40.605 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.605 DEBUG:boto:Data: 2023-09-29T22:22:40.605 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.605 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.605 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.605 DEBUG:boto:Params: {} 2023-09-29T22:22:40.605 DEBUG:boto:Token: None 2023-09-29T22:22:40.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002607430635661c09-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.607 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.607 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.608 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.608 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.608 DEBUG:boto:Method: GET 2023-09-29T22:22:40.608 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.608 DEBUG:boto:Data: 2023-09-29T22:22:40.608 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.608 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.608 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.608 DEBUG:boto:Params: {} 2023-09-29T22:22:40.608 DEBUG:boto:Token: None 2023-09-29T22:22:40.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.609 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2f7138f2a709918-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.611 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.611 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.611 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.611 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.611 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.611 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.611 DEBUG:boto:Data: 2023-09-29T22:22:40.611 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.611 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.611 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.612 DEBUG:boto:Params: {} 2023-09-29T22:22:40.612 DEBUG:boto:Token: None 2023-09-29T22:22:40.612 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.612 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JovsYpny6EU7HKQtnHLwsAMDUS0= 2023-09-29T22:22:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JovsYpny6EU7HKQtnHLwsAMDUS0=', 'Content-Length': '0'} 2023-09-29T22:22:40.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039234f0195a358d6-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.614 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.614 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.614 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.614 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.614 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.614 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.615 DEBUG:boto:Data: 2023-09-29T22:22:40.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.615 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.615 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.615 DEBUG:boto:Params: {} 2023-09-29T22:22:40.615 DEBUG:boto:Token: None 2023-09-29T22:22:40.615 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.615 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JovsYpny6EU7HKQtnHLwsAMDUS0= 2023-09-29T22:22:40.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JovsYpny6EU7HKQtnHLwsAMDUS0=', 'Content-Length': '0'} 2023-09-29T22:22:40.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd47e3fe40769688-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:40.617 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.617 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.617 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.617 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.618 DEBUG:boto:Method: GET 2023-09-29T22:22:40.618 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.618 DEBUG:boto:Data: 2023-09-29T22:22:40.618 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.618 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.618 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.618 DEBUG:boto:Params: {} 2023-09-29T22:22:40.618 DEBUG:boto:Token: None 2023-09-29T22:22:40.618 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.619 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a63e8a3d0ed3e458-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.620 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.620 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.621 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.621 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.621 DEBUG:boto:Method: GET 2023-09-29T22:22:40.621 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.621 DEBUG:boto:Data: 2023-09-29T22:22:40.621 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.621 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.621 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.621 DEBUG:boto:Params: {} 2023-09-29T22:22:40.621 DEBUG:boto:Token: None 2023-09-29T22:22:40.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.622 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f14d28123be860f-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.624 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.624 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.624 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.624 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.624 DEBUG:boto:Method: GET 2023-09-29T22:22:40.624 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.624 DEBUG:boto:Data: 2023-09-29T22:22:40.624 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.624 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.625 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.625 DEBUG:boto:Params: {} 2023-09-29T22:22:40.625 DEBUG:boto:Token: None 2023-09-29T22:22:40.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc0b319415a4066-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.670 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.671 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.671 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.671 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.671 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.671 DEBUG:boto:Method: GET 2023-09-29T22:22:40.671 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.671 DEBUG:boto:Data: 2023-09-29T22:22:40.671 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.672 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.672 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.672 DEBUG:boto:Params: {} 2023-09-29T22:22:40.672 DEBUG:boto:Token: None 2023-09-29T22:22:40.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ed20a65840aabc-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.674 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.675 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.675 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.675 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.675 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:40.675 DEBUG:boto:Data: 2023-09-29T22:22:40.675 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.675 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.675 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.676 DEBUG:boto:Params: {} 2023-09-29T22:22:40.676 DEBUG:boto:Token: None 2023-09-29T22:22:40.676 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy 2023-09-29T22:22:40.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bcp1lj6BGe40YZC0r12m6JlBsXc= 2023-09-29T22:22:40.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bcp1lj6BGe40YZC0r12m6JlBsXc=', 'Content-Length': '0'} 2023-09-29T22:22:40.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000259c9daa83f47ca9-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.678 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.678 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.678 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.678 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:40.678 DEBUG:boto:Data: 2023-09-29T22:22:40.678 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.679 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.679 DEBUG:boto:Params: {} 2023-09-29T22:22:40.679 DEBUG:boto:Token: None 2023-09-29T22:22:40.679 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy 2023-09-29T22:22:40.679 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bcp1lj6BGe40YZC0r12m6JlBsXc= 2023-09-29T22:22:40.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bcp1lj6BGe40YZC0r12m6JlBsXc=', 'Content-Length': '0'} 2023-09-29T22:22:40.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000832714a9e8e10790-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:40.681 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.681 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.681 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.681 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.682 DEBUG:boto:Method: GET 2023-09-29T22:22:40.682 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.682 DEBUG:boto:Data: 2023-09-29T22:22:40.682 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.682 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.682 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.682 DEBUG:boto:Params: {} 2023-09-29T22:22:40.682 DEBUG:boto:Token: None 2023-09-29T22:22:40.682 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.683 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e89841ed25cc0fe-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.684 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:40.684 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:40.685 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.685 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.685 DEBUG:boto:Method: GET 2023-09-29T22:22:40.685 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.685 DEBUG:boto:Data: 2023-09-29T22:22:40.685 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.685 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.685 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.685 DEBUG:boto:Params: {} 2023-09-29T22:22:40.685 DEBUG:boto:Token: None 2023-09-29T22:22:40.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:40.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M= 2023-09-29T22:22:40.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tkhw+HmWf/JL4cBYb6F8YjS0v7M=', 'Content-Length': '0'} 2023-09-29T22:22:40.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a198c6c7e3e68df-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.687 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.687 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.688 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.688 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.688 DEBUG:boto:Method: GET 2023-09-29T22:22:40.688 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.688 DEBUG:boto:Data: 2023-09-29T22:22:40.688 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.688 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.688 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.688 DEBUG:boto:Params: {} 2023-09-29T22:22:40.688 DEBUG:boto:Token: None 2023-09-29T22:22:40.689 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.689 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000910050971356f0ec-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.691 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.691 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.691 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.692 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.692 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.692 DEBUG:boto:Method: GET 2023-09-29T22:22:40.692 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:40.692 DEBUG:boto:Data: 2023-09-29T22:22:40.692 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.692 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.692 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.692 DEBUG:boto:Params: {} 2023-09-29T22:22:40.692 DEBUG:boto:Token: None 2023-09-29T22:22:40.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.693 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003898e96b1ba5f747-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.738 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.739 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.739 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.739 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.739 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.739 DEBUG:boto:Method: GET 2023-09-29T22:22:40.740 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.740 DEBUG:boto:Data: 2023-09-29T22:22:40.740 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.740 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.740 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.740 DEBUG:boto:Params: {} 2023-09-29T22:22:40.740 DEBUG:boto:Token: None 2023-09-29T22:22:40.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.741 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d82cc1ca641e1243-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.742 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.743 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.743 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.743 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.743 DEBUG:boto:Method: GET 2023-09-29T22:22:40.743 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.743 DEBUG:boto:Data: 2023-09-29T22:22:40.743 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.743 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.743 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.743 DEBUG:boto:Params: {} 2023-09-29T22:22:40.744 DEBUG:boto:Token: None 2023-09-29T22:22:40.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.744 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8ffb8fd3597aaab-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.746 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.746 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.746 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.746 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.746 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.746 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.747 DEBUG:boto:Data: 2023-09-29T22:22:40.747 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.747 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.747 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.747 DEBUG:boto:Params: {} 2023-09-29T22:22:40.747 DEBUG:boto:Token: None 2023-09-29T22:22:40.747 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.747 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iAo2FmYUMWWFI+0yE1nhpjNw4FQ= 2023-09-29T22:22:40.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iAo2FmYUMWWFI+0yE1nhpjNw4FQ=', 'Content-Length': '0'} 2023-09-29T22:22:40.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000179e0114ae3a68f2-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.749 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.749 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.749 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.749 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.749 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.750 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.750 DEBUG:boto:Data: 2023-09-29T22:22:40.750 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.750 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.750 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.750 DEBUG:boto:Params: {} 2023-09-29T22:22:40.750 DEBUG:boto:Token: None 2023-09-29T22:22:40.750 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iAo2FmYUMWWFI+0yE1nhpjNw4FQ= 2023-09-29T22:22:40.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iAo2FmYUMWWFI+0yE1nhpjNw4FQ=', 'Content-Length': '0'} 2023-09-29T22:22:40.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e36806d3f7a9fcc-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.752 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.752 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.753 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.753 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.753 DEBUG:boto:Method: GET 2023-09-29T22:22:40.753 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.753 DEBUG:boto:Data: 2023-09-29T22:22:40.753 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.753 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.753 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.753 DEBUG:boto:Params: {} 2023-09-29T22:22:40.753 DEBUG:boto:Token: None 2023-09-29T22:22:40.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.754 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e85e7c3484e28529-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.756 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.756 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.756 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.756 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.756 DEBUG:boto:Method: GET 2023-09-29T22:22:40.756 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.756 DEBUG:boto:Data: 2023-09-29T22:22:40.757 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.757 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.757 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.757 DEBUG:boto:Params: {} 2023-09-29T22:22:40.757 DEBUG:boto:Token: None 2023-09-29T22:22:40.757 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.757 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be29427894f3f239-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.759 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.759 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.759 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.759 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.760 DEBUG:boto:Method: GET 2023-09-29T22:22:40.760 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.760 DEBUG:boto:Data: 2023-09-29T22:22:40.760 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.760 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.760 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.760 DEBUG:boto:Params: {} 2023-09-29T22:22:40.760 DEBUG:boto:Token: None 2023-09-29T22:22:40.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.761 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008766eea8cfe38ca2-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.763 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.763 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.764 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.764 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.764 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.764 DEBUG:boto:Method: GET 2023-09-29T22:22:40.764 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.764 DEBUG:boto:Data: 2023-09-29T22:22:40.764 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.764 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.764 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.764 DEBUG:boto:Params: {} 2023-09-29T22:22:40.765 DEBUG:boto:Token: None 2023-09-29T22:22:40.765 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.765 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007169dedef5a84609-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.767 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.767 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.767 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.768 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.768 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:40.768 DEBUG:boto:Data: 2023-09-29T22:22:40.768 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.768 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.768 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.768 DEBUG:boto:Params: {} 2023-09-29T22:22:40.768 DEBUG:boto:Token: None 2023-09-29T22:22:40.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj 2023-09-29T22:22:40.769 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU= 2023-09-29T22:22:40.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU=', 'Content-Length': '0'} 2023-09-29T22:22:40.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3ce10288bd8be8a-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.771 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.772 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.772 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.772 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:40.772 DEBUG:boto:Data: 2023-09-29T22:22:40.772 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.772 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.772 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.772 DEBUG:boto:Params: {} 2023-09-29T22:22:40.772 DEBUG:boto:Token: None 2023-09-29T22:22:40.773 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj 2023-09-29T22:22:40.773 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU= 2023-09-29T22:22:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU=', 'Content-Length': '0'} 2023-09-29T22:22:40.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044c5011eb0863b73-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.776 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.776 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.776 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.776 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.776 DEBUG:boto:Method: GET 2023-09-29T22:22:40.776 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.776 DEBUG:boto:Data: 2023-09-29T22:22:40.776 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.776 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.777 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.777 DEBUG:boto:Params: {} 2023-09-29T22:22:40.777 DEBUG:boto:Token: None 2023-09-29T22:22:40.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.777 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ed2044020e40679-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.822 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.823 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.823 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.823 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.823 DEBUG:boto:Method: GET 2023-09-29T22:22:40.823 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.823 DEBUG:boto:Data: 2023-09-29T22:22:40.823 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.823 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.823 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.824 DEBUG:boto:Params: {} 2023-09-29T22:22:40.824 DEBUG:boto:Token: None 2023-09-29T22:22:40.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000123f9f8bd50916d6-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.826 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.826 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.827 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.827 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.827 DEBUG:boto:Method: GET 2023-09-29T22:22:40.827 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.827 DEBUG:boto:Data: 2023-09-29T22:22:40.827 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.827 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.827 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.827 DEBUG:boto:Params: {} 2023-09-29T22:22:40.827 DEBUG:boto:Token: None 2023-09-29T22:22:40.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3894af1070d7d2-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.830 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.831 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.831 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.831 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.831 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.831 DEBUG:boto:Method: GET 2023-09-29T22:22:40.831 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.831 DEBUG:boto:Data: 2023-09-29T22:22:40.832 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.832 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.832 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.832 DEBUG:boto:Params: {} 2023-09-29T22:22:40.832 DEBUG:boto:Token: None 2023-09-29T22:22:40.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b378201fd92c6e87-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.835 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.835 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.835 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.836 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.836 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.836 DEBUG:boto:Method: GET 2023-09-29T22:22:40.836 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.836 DEBUG:boto:Data: 2023-09-29T22:22:40.836 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.836 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.836 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.836 DEBUG:boto:Params: {} 2023-09-29T22:22:40.836 DEBUG:boto:Token: None 2023-09-29T22:22:40.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.837 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs= 2023-09-29T22:22:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs=', 'Content-Length': '0'} 2023-09-29T22:22:40.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000404759a2177f175b-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.839 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.839 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.839 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.839 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.839 DEBUG:boto:Method: GET 2023-09-29T22:22:40.839 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.839 DEBUG:boto:Data: 2023-09-29T22:22:40.839 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.840 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.840 DEBUG:boto:Params: {} 2023-09-29T22:22:40.840 DEBUG:boto:Token: None 2023-09-29T22:22:40.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs= 2023-09-29T22:22:40.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs=', 'Content-Length': '0'} 2023-09-29T22:22:40.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022106e63413d37da-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.842 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.842 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.842 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.842 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.842 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.842 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.842 DEBUG:boto:Data: 2023-09-29T22:22:40.843 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.843 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.843 DEBUG:boto:Params: {} 2023-09-29T22:22:40.843 DEBUG:boto:Token: None 2023-09-29T22:22:40.843 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.843 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ww8DLkltAqI1KoDXYhudHpTG8NI= 2023-09-29T22:22:40.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ww8DLkltAqI1KoDXYhudHpTG8NI=', 'Content-Length': '0'} 2023-09-29T22:22:40.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006946e3ca350fea9e-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.845 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.845 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.845 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.845 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.846 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.846 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.846 DEBUG:boto:Data: 2023-09-29T22:22:40.846 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.846 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.846 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.846 DEBUG:boto:Params: {} 2023-09-29T22:22:40.846 DEBUG:boto:Token: None 2023-09-29T22:22:40.846 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ww8DLkltAqI1KoDXYhudHpTG8NI= 2023-09-29T22:22:40.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ww8DLkltAqI1KoDXYhudHpTG8NI=', 'Content-Length': '0'} 2023-09-29T22:22:40.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a1832a3ad06cbd-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.849 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.849 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.849 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.849 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.849 DEBUG:boto:Method: GET 2023-09-29T22:22:40.849 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.849 DEBUG:boto:Data: 2023-09-29T22:22:40.849 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.850 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.850 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.850 DEBUG:boto:Params: {} 2023-09-29T22:22:40.850 DEBUG:boto:Token: None 2023-09-29T22:22:40.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.850 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs= 2023-09-29T22:22:40.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs=', 'Content-Length': '0'} 2023-09-29T22:22:40.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a95e680b448a6f18-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.894 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.894 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.895 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.895 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.895 DEBUG:boto:Method: GET 2023-09-29T22:22:40.895 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.895 DEBUG:boto:Data: 2023-09-29T22:22:40.895 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.895 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.895 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.895 DEBUG:boto:Params: {} 2023-09-29T22:22:40.895 DEBUG:boto:Token: None 2023-09-29T22:22:40.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:40.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs= 2023-09-29T22:22:40.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qa7SKxXmz4S141xzMpbyWZiPPAs=', 'Content-Length': '0'} 2023-09-29T22:22:40.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008587436c509c15b9-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.938 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.939 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.939 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.939 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.939 DEBUG:boto:Method: GET 2023-09-29T22:22:40.939 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.939 DEBUG:boto:Data: 2023-09-29T22:22:40.939 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.939 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.939 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.940 DEBUG:boto:Params: {} 2023-09-29T22:22:40.940 DEBUG:boto:Token: None 2023-09-29T22:22:40.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562d64a423a06265-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.942 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.943 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:40.943 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:40.943 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.943 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.943 DEBUG:boto:Method: GET 2023-09-29T22:22:40.943 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:40.943 DEBUG:boto:Data: 2023-09-29T22:22:40.944 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.944 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.944 DEBUG:boto:Params: {} 2023-09-29T22:22:40.944 DEBUG:boto:Token: None 2023-09-29T22:22:40.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/ 2023-09-29T22:22:40.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA= 2023-09-29T22:22:40.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JVhtKP0HDujSBMulh49cKvlTpFA=', 'Content-Length': '0'} 2023-09-29T22:22:40.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004982d3aab222c0c8-0065174e30-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.946 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:40.947 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.947 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.947 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.947 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:40.947 DEBUG:boto:Data: 2023-09-29T22:22:40.947 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.947 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.947 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.948 DEBUG:boto:Params: {} 2023-09-29T22:22:40.948 DEBUG:boto:Token: None 2023-09-29T22:22:40.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj 2023-09-29T22:22:40.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU= 2023-09-29T22:22:40.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU=', 'Content-Length': '0'} 2023-09-29T22:22:40.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d6e812aee0bad73-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.951 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.951 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.951 DEBUG:boto:Method: HEAD 2023-09-29T22:22:40.951 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:40.951 DEBUG:boto:Data: 2023-09-29T22:22:40.951 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.952 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.952 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.952 DEBUG:boto:Params: {} 2023-09-29T22:22:40.952 DEBUG:boto:Token: None 2023-09-29T22:22:40.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj 2023-09-29T22:22:40.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU= 2023-09-29T22:22:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PvtKaVYgDkJo7ySPhNeWM8DfKU=', 'Content-Length': '0'} 2023-09-29T22:22:40.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002eab8858445b249a-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:40.955 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.955 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.955 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.955 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.955 DEBUG:boto:Method: GET 2023-09-29T22:22:40.955 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.956 DEBUG:boto:Data: 2023-09-29T22:22:40.956 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.956 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:40.956 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.956 DEBUG:boto:Params: {} 2023-09-29T22:22:40.956 DEBUG:boto:Token: None 2023-09-29T22:22:40.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045ddfee54b9c2b74-0065174e30-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:40.958 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:40.959 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:40.959 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.959 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.959 DEBUG:boto:Method: GET 2023-09-29T22:22:40.959 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.959 DEBUG:boto:Data: 2023-09-29T22:22:40.959 DEBUG:boto:Headers: {} 2023-09-29T22:22:40.959 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:40.959 DEBUG:boto:Port: 8000 2023-09-29T22:22:40.959 DEBUG:boto:Params: {} 2023-09-29T22:22:40.960 DEBUG:boto:Token: None 2023-09-29T22:22:40.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:40 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:40.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8= 2023-09-29T22:22:40.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:40 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbyVZXW0dVlKh2O2rJOPfaF3BW8=', 'Content-Length': '0'} 2023-09-29T22:22:40.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbe16485459a7f0f-0065174e30-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.006 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.007 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.007 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.007 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.007 DEBUG:boto:Method: GET 2023-09-29T22:22:41.007 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.007 DEBUG:boto:Data: 2023-09-29T22:22:41.007 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.007 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.008 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.008 DEBUG:boto:Params: {} 2023-09-29T22:22:41.008 DEBUG:boto:Token: None 2023-09-29T22:22:41.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.008 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081abc2e1e0620a3b-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.013 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.014 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.014 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.014 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.014 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.014 DEBUG:boto:Method: GET 2023-09-29T22:22:41.014 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.015 DEBUG:boto:Data: 2023-09-29T22:22:41.015 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.015 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.015 DEBUG:boto:Params: {} 2023-09-29T22:22:41.015 DEBUG:boto:Token: None 2023-09-29T22:22:41.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326f289975e8ffe6-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.018 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.018 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.018 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.018 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.019 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.019 DEBUG:boto:Method: GET 2023-09-29T22:22:41.019 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.019 DEBUG:boto:Data: 2023-09-29T22:22:41.019 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.019 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.019 DEBUG:boto:Params: {} 2023-09-29T22:22:41.019 DEBUG:boto:Token: None 2023-09-29T22:22:41.021 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.021 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA= 2023-09-29T22:22:41.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA=', 'Content-Length': '0'} 2023-09-29T22:22:41.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b19497d3dea8229f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.070 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.071 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.071 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.071 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.071 DEBUG:boto:Method: GET 2023-09-29T22:22:41.071 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.071 DEBUG:boto:Data: 2023-09-29T22:22:41.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.071 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.071 DEBUG:boto:Params: {} 2023-09-29T22:22:41.072 DEBUG:boto:Token: None 2023-09-29T22:22:41.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA= 2023-09-29T22:22:41.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA=', 'Content-Length': '0'} 2023-09-29T22:22:41.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e35550025e9de1e5-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.074 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.074 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.074 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.074 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.074 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.075 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.075 DEBUG:boto:Data: 2023-09-29T22:22:41.075 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.075 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.075 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.075 DEBUG:boto:Params: {} 2023-09-29T22:22:41.075 DEBUG:boto:Token: None 2023-09-29T22:22:41.075 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VxEmjtMxi5kY7eTVOw1LV8YD0rY= 2023-09-29T22:22:41.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VxEmjtMxi5kY7eTVOw1LV8YD0rY=', 'Content-Length': '0'} 2023-09-29T22:22:41.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001abbac014344da01-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.077 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.078 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.078 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.078 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.078 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.078 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.078 DEBUG:boto:Data: 2023-09-29T22:22:41.078 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.078 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.078 DEBUG:boto:Params: {} 2023-09-29T22:22:41.079 DEBUG:boto:Token: None 2023-09-29T22:22:41.079 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.079 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VxEmjtMxi5kY7eTVOw1LV8YD0rY= 2023-09-29T22:22:41.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VxEmjtMxi5kY7eTVOw1LV8YD0rY=', 'Content-Length': '0'} 2023-09-29T22:22:41.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0a72324f74e8806-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.081 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.081 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.081 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.081 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.081 DEBUG:boto:Method: GET 2023-09-29T22:22:41.081 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.081 DEBUG:boto:Data: 2023-09-29T22:22:41.081 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.082 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.082 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.082 DEBUG:boto:Params: {} 2023-09-29T22:22:41.082 DEBUG:boto:Token: None 2023-09-29T22:22:41.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA= 2023-09-29T22:22:41.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA=', 'Content-Length': '0'} 2023-09-29T22:22:41.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e44711b65ab7dcc-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.084 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.084 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.084 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.085 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.085 DEBUG:boto:Method: GET 2023-09-29T22:22:41.085 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.085 DEBUG:boto:Data: 2023-09-29T22:22:41.085 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.085 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.085 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.085 DEBUG:boto:Params: {} 2023-09-29T22:22:41.085 DEBUG:boto:Token: None 2023-09-29T22:22:41.085 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:41.086 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA= 2023-09-29T22:22:41.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ofprNv19idUx4ouYHDEudHa5bxA=', 'Content-Length': '0'} 2023-09-29T22:22:41.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c2788283e1c2edd-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.087 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.088 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.088 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.088 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.088 DEBUG:boto:Method: GET 2023-09-29T22:22:41.088 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.088 DEBUG:boto:Data: 2023-09-29T22:22:41.088 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.088 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.088 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.089 DEBUG:boto:Params: {} 2023-09-29T22:22:41.089 DEBUG:boto:Token: None 2023-09-29T22:22:41.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.089 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab33e0d30929e81-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.092 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.092 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.092 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.092 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.093 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.093 DEBUG:boto:Method: GET 2023-09-29T22:22:41.093 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.093 DEBUG:boto:Data: 2023-09-29T22:22:41.093 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.093 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.093 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.093 DEBUG:boto:Params: {} 2023-09-29T22:22:41.093 DEBUG:boto:Token: None 2023-09-29T22:22:41.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.094 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ea2538f0bb46a1-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.096 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.097 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.097 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.097 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.097 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.097 DEBUG:boto:Data: 2023-09-29T22:22:41.097 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.097 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.097 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.097 DEBUG:boto:Params: {} 2023-09-29T22:22:41.097 DEBUG:boto:Token: None 2023-09-29T22:22:41.098 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.098 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000246a8a860baf4562-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.101 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.101 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.101 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.101 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.101 DEBUG:boto:Data: 2023-09-29T22:22:41.101 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.101 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.101 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.101 DEBUG:boto:Params: {} 2023-09-29T22:22:41.102 DEBUG:boto:Token: None 2023-09-29T22:22:41.102 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.102 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.104 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063bd19a318c980aa-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.105 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.105 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.105 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.105 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.105 DEBUG:boto:Method: GET 2023-09-29T22:22:41.105 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.105 DEBUG:boto:Data: 2023-09-29T22:22:41.105 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.105 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.105 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.105 DEBUG:boto:Params: {} 2023-09-29T22:22:41.106 DEBUG:boto:Token: None 2023-09-29T22:22:41.106 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.106 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000524353303b0c0aa-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.108 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.108 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.108 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.108 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.108 DEBUG:boto:Method: GET 2023-09-29T22:22:41.108 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.108 DEBUG:boto:Data: 2023-09-29T22:22:41.109 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.109 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.109 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.109 DEBUG:boto:Params: {} 2023-09-29T22:22:41.109 DEBUG:boto:Token: None 2023-09-29T22:22:41.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.109 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003239a66e84d55e41-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.111 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.111 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.111 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.111 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.112 DEBUG:boto:Method: GET 2023-09-29T22:22:41.112 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.112 DEBUG:boto:Data: 2023-09-29T22:22:41.112 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.112 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.112 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.112 DEBUG:boto:Params: {} 2023-09-29T22:22:41.112 DEBUG:boto:Token: None 2023-09-29T22:22:41.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.113 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab55a1a0d878d879-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.115 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.115 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.116 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.116 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.116 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.116 DEBUG:boto:Method: GET 2023-09-29T22:22:41.116 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.116 DEBUG:boto:Data: 2023-09-29T22:22:41.116 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.116 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.116 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.116 DEBUG:boto:Params: {} 2023-09-29T22:22:41.117 DEBUG:boto:Token: None 2023-09-29T22:22:41.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.117 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099017df77de67887-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.119 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.120 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.120 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.120 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.120 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.120 DEBUG:boto:Method: GET 2023-09-29T22:22:41.120 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.120 DEBUG:boto:Data: 2023-09-29T22:22:41.120 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.120 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.121 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.121 DEBUG:boto:Params: {} 2023-09-29T22:22:41.121 DEBUG:boto:Token: None 2023-09-29T22:22:41.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY= 2023-09-29T22:22:41.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY=', 'Content-Length': '0'} 2023-09-29T22:22:41.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a414a351fd940f5-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.170 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.171 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.171 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.171 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.171 DEBUG:boto:Method: GET 2023-09-29T22:22:41.171 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.171 DEBUG:boto:Data: 2023-09-29T22:22:41.171 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.171 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.171 DEBUG:boto:Params: {} 2023-09-29T22:22:41.172 DEBUG:boto:Token: None 2023-09-29T22:22:41.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY= 2023-09-29T22:22:41.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY=', 'Content-Length': '0'} 2023-09-29T22:22:41.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000393b0dd2c325b5fe-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.174 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.174 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.174 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.174 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.174 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.174 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.174 DEBUG:boto:Data: 2023-09-29T22:22:41.175 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.175 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.175 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.175 DEBUG:boto:Params: {} 2023-09-29T22:22:41.175 DEBUG:boto:Token: None 2023-09-29T22:22:41.175 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:79ViVcsC7WM002fb/n9p4VLEd3A= 2023-09-29T22:22:41.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:79ViVcsC7WM002fb/n9p4VLEd3A=', 'Content-Length': '0'} 2023-09-29T22:22:41.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064ecbd15513216d5-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.177 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.177 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.177 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.178 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.178 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.178 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.178 DEBUG:boto:Data: 2023-09-29T22:22:41.178 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.178 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.178 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.178 DEBUG:boto:Params: {} 2023-09-29T22:22:41.178 DEBUG:boto:Token: None 2023-09-29T22:22:41.178 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:79ViVcsC7WM002fb/n9p4VLEd3A= 2023-09-29T22:22:41.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:79ViVcsC7WM002fb/n9p4VLEd3A=', 'Content-Length': '0'} 2023-09-29T22:22:41.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000944d29755a878020-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.180 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.181 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.181 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.181 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.181 DEBUG:boto:Method: GET 2023-09-29T22:22:41.181 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.181 DEBUG:boto:Data: 2023-09-29T22:22:41.181 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.181 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.181 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.181 DEBUG:boto:Params: {} 2023-09-29T22:22:41.182 DEBUG:boto:Token: None 2023-09-29T22:22:41.182 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.182 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY= 2023-09-29T22:22:41.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY=', 'Content-Length': '0'} 2023-09-29T22:22:41.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d41ddcba9bd2a5-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.184 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.184 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.184 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.184 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.184 DEBUG:boto:Method: GET 2023-09-29T22:22:41.184 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.184 DEBUG:boto:Data: 2023-09-29T22:22:41.185 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.185 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.185 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.185 DEBUG:boto:Params: {} 2023-09-29T22:22:41.185 DEBUG:boto:Token: None 2023-09-29T22:22:41.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:41.185 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY= 2023-09-29T22:22:41.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RsT/V2Ht1aFecHnlWJJhbmA9yTY=', 'Content-Length': '0'} 2023-09-29T22:22:41.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000663427f23b861307-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.187 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.187 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.187 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.188 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.188 DEBUG:boto:Method: GET 2023-09-29T22:22:41.188 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.188 DEBUG:boto:Data: 2023-09-29T22:22:41.188 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.188 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.188 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.188 DEBUG:boto:Params: {} 2023-09-29T22:22:41.188 DEBUG:boto:Token: None 2023-09-29T22:22:41.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.189 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001077df6819f91e9a-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.191 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.191 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.191 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.192 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.192 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.192 DEBUG:boto:Method: GET 2023-09-29T22:22:41.192 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.192 DEBUG:boto:Data: 2023-09-29T22:22:41.192 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.192 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.192 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.192 DEBUG:boto:Params: {} 2023-09-29T22:22:41.192 DEBUG:boto:Token: None 2023-09-29T22:22:41.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e813de36c077df70-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.238 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.239 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.239 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.239 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.239 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.239 DEBUG:boto:Data: 2023-09-29T22:22:41.239 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.239 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.239 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.239 DEBUG:boto:Params: {} 2023-09-29T22:22:41.240 DEBUG:boto:Token: None 2023-09-29T22:22:41.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000048aef076879f354-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.243 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.243 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.243 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.243 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.243 DEBUG:boto:Data: 2023-09-29T22:22:41.243 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.243 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.244 DEBUG:boto:Params: {} 2023-09-29T22:22:41.244 DEBUG:boto:Token: None 2023-09-29T22:22:41.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f2b69f9b9c7f1e3-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.247 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.247 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.247 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.247 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.247 DEBUG:boto:Method: GET 2023-09-29T22:22:41.248 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.248 DEBUG:boto:Data: 2023-09-29T22:22:41.248 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.248 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.248 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.248 DEBUG:boto:Params: {} 2023-09-29T22:22:41.248 DEBUG:boto:Token: None 2023-09-29T22:22:41.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000788264388efba5af-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.251 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.251 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.251 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.251 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.251 DEBUG:boto:Method: GET 2023-09-29T22:22:41.251 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.251 DEBUG:boto:Data: 2023-09-29T22:22:41.251 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.251 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.251 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.252 DEBUG:boto:Params: {} 2023-09-29T22:22:41.252 DEBUG:boto:Token: None 2023-09-29T22:22:41.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034db718f928bfb7e-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.254 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.254 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.254 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.254 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.254 DEBUG:boto:Method: GET 2023-09-29T22:22:41.255 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.255 DEBUG:boto:Data: 2023-09-29T22:22:41.255 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.255 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.255 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.255 DEBUG:boto:Params: {} 2023-09-29T22:22:41.255 DEBUG:boto:Token: None 2023-09-29T22:22:41.255 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.255 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f199bf230343f312-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.302 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.303 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.303 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.303 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.304 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.304 DEBUG:boto:Method: GET 2023-09-29T22:22:41.304 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.304 DEBUG:boto:Data: 2023-09-29T22:22:41.304 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.305 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.305 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.305 DEBUG:boto:Params: {} 2023-09-29T22:22:41.305 DEBUG:boto:Token: None 2023-09-29T22:22:41.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.306 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e83cf4a252d19ed-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.309 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.310 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.310 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.310 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.310 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.311 DEBUG:boto:Method: GET 2023-09-29T22:22:41.311 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.311 DEBUG:boto:Data: 2023-09-29T22:22:41.311 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.311 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.312 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.312 DEBUG:boto:Params: {} 2023-09-29T22:22:41.312 DEBUG:boto:Token: None 2023-09-29T22:22:41.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg= 2023-09-29T22:22:41.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg=', 'Content-Length': '0'} 2023-09-29T22:22:41.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090adb49c35a46909-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.315 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.315 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.315 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.315 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.315 DEBUG:boto:Method: GET 2023-09-29T22:22:41.315 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.315 DEBUG:boto:Data: 2023-09-29T22:22:41.315 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.316 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.316 DEBUG:boto:Params: {} 2023-09-29T22:22:41.316 DEBUG:boto:Token: None 2023-09-29T22:22:41.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg= 2023-09-29T22:22:41.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg=', 'Content-Length': '0'} 2023-09-29T22:22:41.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b82dfe6ec6812332-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.318 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.318 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.318 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.318 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.319 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.319 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.319 DEBUG:boto:Data: 2023-09-29T22:22:41.319 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.319 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.319 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.319 DEBUG:boto:Params: {} 2023-09-29T22:22:41.319 DEBUG:boto:Token: None 2023-09-29T22:22:41.319 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DduNexTTfIAxUUtI93xKXLnDRk0= 2023-09-29T22:22:41.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DduNexTTfIAxUUtI93xKXLnDRk0=', 'Content-Length': '0'} 2023-09-29T22:22:41.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e706194266b413d4-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.322 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.322 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.322 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.322 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.322 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.322 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.323 DEBUG:boto:Data: 2023-09-29T22:22:41.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.323 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.323 DEBUG:boto:Params: {} 2023-09-29T22:22:41.323 DEBUG:boto:Token: None 2023-09-29T22:22:41.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DduNexTTfIAxUUtI93xKXLnDRk0= 2023-09-29T22:22:41.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DduNexTTfIAxUUtI93xKXLnDRk0=', 'Content-Length': '0'} 2023-09-29T22:22:41.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047288b31d0f3e064-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.325 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.325 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.326 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.326 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.326 DEBUG:boto:Method: GET 2023-09-29T22:22:41.326 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.326 DEBUG:boto:Data: 2023-09-29T22:22:41.326 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.326 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.326 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.326 DEBUG:boto:Params: {} 2023-09-29T22:22:41.326 DEBUG:boto:Token: None 2023-09-29T22:22:41.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg= 2023-09-29T22:22:41.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg=', 'Content-Length': '0'} 2023-09-29T22:22:41.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a477073af31dfdf1-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.329 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.329 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.329 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.329 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.329 DEBUG:boto:Method: GET 2023-09-29T22:22:41.329 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.329 DEBUG:boto:Data: 2023-09-29T22:22:41.329 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.330 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.330 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.330 DEBUG:boto:Params: {} 2023-09-29T22:22:41.330 DEBUG:boto:Token: None 2023-09-29T22:22:41.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:41.330 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg= 2023-09-29T22:22:41.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jr5q2BAroQ7kCOuTdN+Ms/tJegg=', 'Content-Length': '0'} 2023-09-29T22:22:41.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dc47426023a5b59-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.332 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.332 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.332 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.333 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.333 DEBUG:boto:Method: GET 2023-09-29T22:22:41.333 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.333 DEBUG:boto:Data: 2023-09-29T22:22:41.333 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.333 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.333 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.333 DEBUG:boto:Params: {} 2023-09-29T22:22:41.333 DEBUG:boto:Token: None 2023-09-29T22:22:41.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.334 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000407afed193f425cb-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.336 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.336 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.337 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.337 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.337 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.337 DEBUG:boto:Method: GET 2023-09-29T22:22:41.337 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.337 DEBUG:boto:Data: 2023-09-29T22:22:41.337 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.337 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.337 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.337 DEBUG:boto:Params: {} 2023-09-29T22:22:41.338 DEBUG:boto:Token: None 2023-09-29T22:22:41.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c8d18839467883-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.340 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.340 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.341 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.341 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.341 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.341 DEBUG:boto:Data: 2023-09-29T22:22:41.341 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.341 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.341 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.341 DEBUG:boto:Params: {} 2023-09-29T22:22:41.341 DEBUG:boto:Token: None 2023-09-29T22:22:41.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e8c4e138082dcb5-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.345 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.345 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.345 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.345 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.345 DEBUG:boto:Data: 2023-09-29T22:22:41.345 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.345 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.345 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.346 DEBUG:boto:Params: {} 2023-09-29T22:22:41.346 DEBUG:boto:Token: None 2023-09-29T22:22:41.346 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.346 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff5a2e6875d4bbc4-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.349 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.349 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.349 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.349 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.349 DEBUG:boto:Method: GET 2023-09-29T22:22:41.349 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.349 DEBUG:boto:Data: 2023-09-29T22:22:41.350 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.350 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.350 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.350 DEBUG:boto:Params: {} 2023-09-29T22:22:41.350 DEBUG:boto:Token: None 2023-09-29T22:22:41.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ab3665de54987f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.353 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.353 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.353 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.353 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.353 DEBUG:boto:Method: GET 2023-09-29T22:22:41.353 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.353 DEBUG:boto:Data: 2023-09-29T22:22:41.353 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.354 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.354 DEBUG:boto:Params: {} 2023-09-29T22:22:41.354 DEBUG:boto:Token: None 2023-09-29T22:22:41.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.354 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051917cbc06e39c9b-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.356 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.356 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.356 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.356 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.356 DEBUG:boto:Method: GET 2023-09-29T22:22:41.356 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.356 DEBUG:boto:Data: 2023-09-29T22:22:41.357 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.357 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.357 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.357 DEBUG:boto:Params: {} 2023-09-29T22:22:41.357 DEBUG:boto:Token: None 2023-09-29T22:22:41.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011310a0b1050db76-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.360 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.360 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.360 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.361 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.361 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.361 DEBUG:boto:Method: GET 2023-09-29T22:22:41.361 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.361 DEBUG:boto:Data: 2023-09-29T22:22:41.361 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.361 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.361 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.361 DEBUG:boto:Params: {} 2023-09-29T22:22:41.361 DEBUG:boto:Token: None 2023-09-29T22:22:41.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.362 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306bad5a98d6209f-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.364 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.364 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.364 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.365 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.365 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.365 DEBUG:boto:Method: GET 2023-09-29T22:22:41.365 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.365 DEBUG:boto:Data: 2023-09-29T22:22:41.365 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.365 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.365 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.365 DEBUG:boto:Params: {} 2023-09-29T22:22:41.365 DEBUG:boto:Token: None 2023-09-29T22:22:41.366 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.366 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ= 2023-09-29T22:22:41.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df181452cf5fce51-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.410 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.411 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.411 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.411 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.411 DEBUG:boto:Method: GET 2023-09-29T22:22:41.411 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.411 DEBUG:boto:Data: 2023-09-29T22:22:41.411 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.411 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.411 DEBUG:boto:Params: {} 2023-09-29T22:22:41.412 DEBUG:boto:Token: None 2023-09-29T22:22:41.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ= 2023-09-29T22:22:41.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ebe157fe7f4813f-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.414 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.414 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.414 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.414 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.414 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.414 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.414 DEBUG:boto:Data: 2023-09-29T22:22:41.414 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.415 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.415 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.415 DEBUG:boto:Params: {} 2023-09-29T22:22:41.415 DEBUG:boto:Token: None 2023-09-29T22:22:41.415 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rU9hdugx6bYDz7W9KQB9N86987g= 2023-09-29T22:22:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rU9hdugx6bYDz7W9KQB9N86987g=', 'Content-Length': '0'} 2023-09-29T22:22:41.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009210c1a8cbf3d738-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.417 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.417 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.417 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.417 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.418 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.418 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.418 DEBUG:boto:Data: 2023-09-29T22:22:41.418 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.418 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.418 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.418 DEBUG:boto:Params: {} 2023-09-29T22:22:41.418 DEBUG:boto:Token: None 2023-09-29T22:22:41.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rU9hdugx6bYDz7W9KQB9N86987g= 2023-09-29T22:22:41.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rU9hdugx6bYDz7W9KQB9N86987g=', 'Content-Length': '0'} 2023-09-29T22:22:41.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072f85eb6d444c412-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.423 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.423 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.423 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.423 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.424 DEBUG:boto:Method: GET 2023-09-29T22:22:41.424 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.424 DEBUG:boto:Data: 2023-09-29T22:22:41.424 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.424 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.424 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.424 DEBUG:boto:Params: {} 2023-09-29T22:22:41.424 DEBUG:boto:Token: None 2023-09-29T22:22:41.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ= 2023-09-29T22:22:41.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d91ec4a201872c06-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.427 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.427 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.427 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.427 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.427 DEBUG:boto:Method: GET 2023-09-29T22:22:41.427 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.427 DEBUG:boto:Data: 2023-09-29T22:22:41.427 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.427 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.428 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.428 DEBUG:boto:Params: {} 2023-09-29T22:22:41.428 DEBUG:boto:Token: None 2023-09-29T22:22:41.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:41.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ= 2023-09-29T22:22:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ToCQxrxKkmq+z5zpEYu+2LI7iyQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002852d3e88a61e75d-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.474 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.474 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.475 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.475 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.475 DEBUG:boto:Method: GET 2023-09-29T22:22:41.475 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.475 DEBUG:boto:Data: 2023-09-29T22:22:41.475 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.475 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.475 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.475 DEBUG:boto:Params: {} 2023-09-29T22:22:41.475 DEBUG:boto:Token: None 2023-09-29T22:22:41.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001acb7ee9e0c4cac6-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.478 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.479 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.479 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.479 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.479 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.479 DEBUG:boto:Method: GET 2023-09-29T22:22:41.479 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.479 DEBUG:boto:Data: 2023-09-29T22:22:41.480 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.480 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.480 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.480 DEBUG:boto:Params: {} 2023-09-29T22:22:41.480 DEBUG:boto:Token: None 2023-09-29T22:22:41.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005025006499e6e35a-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.483 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.483 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.483 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.483 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.483 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.483 DEBUG:boto:Data: 2023-09-29T22:22:41.483 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.484 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.484 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.484 DEBUG:boto:Params: {} 2023-09-29T22:22:41.484 DEBUG:boto:Token: None 2023-09-29T22:22:41.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002674bf835bc00f9b-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.487 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.487 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.488 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.488 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.488 DEBUG:boto:Data: 2023-09-29T22:22:41.488 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.488 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.488 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.488 DEBUG:boto:Params: {} 2023-09-29T22:22:41.488 DEBUG:boto:Token: None 2023-09-29T22:22:41.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aeeb0836d52adc9-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.491 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.492 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.492 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.492 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.492 DEBUG:boto:Method: GET 2023-09-29T22:22:41.492 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.492 DEBUG:boto:Data: 2023-09-29T22:22:41.492 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.492 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.492 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.492 DEBUG:boto:Params: {} 2023-09-29T22:22:41.492 DEBUG:boto:Token: None 2023-09-29T22:22:41.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.493 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b997aa2441eaa79c-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.538 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.538 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.538 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.539 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.539 DEBUG:boto:Method: GET 2023-09-29T22:22:41.539 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.539 DEBUG:boto:Data: 2023-09-29T22:22:41.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.539 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.539 DEBUG:boto:Params: {} 2023-09-29T22:22:41.539 DEBUG:boto:Token: None 2023-09-29T22:22:41.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6704717b0d9a3db-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.542 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.542 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.542 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.542 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.542 DEBUG:boto:Method: GET 2023-09-29T22:22:41.542 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.542 DEBUG:boto:Data: 2023-09-29T22:22:41.542 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.542 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.542 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.543 DEBUG:boto:Params: {} 2023-09-29T22:22:41.543 DEBUG:boto:Token: None 2023-09-29T22:22:41.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.543 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534b7c9f349900a4-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.545 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.546 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.546 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.546 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.546 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.546 DEBUG:boto:Method: GET 2023-09-29T22:22:41.546 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.546 DEBUG:boto:Data: 2023-09-29T22:22:41.546 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.547 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.547 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.547 DEBUG:boto:Params: {} 2023-09-29T22:22:41.547 DEBUG:boto:Token: None 2023-09-29T22:22:41.547 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.547 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcbca8ec58fd4cc5-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.550 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.550 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.550 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.551 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.551 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.551 DEBUG:boto:Method: GET 2023-09-29T22:22:41.551 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.551 DEBUG:boto:Data: 2023-09-29T22:22:41.551 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.551 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.551 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.551 DEBUG:boto:Params: {} 2023-09-29T22:22:41.551 DEBUG:boto:Token: None 2023-09-29T22:22:41.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw= 2023-09-29T22:22:41.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw=', 'Content-Length': '0'} 2023-09-29T22:22:41.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cf87cb6c7a77a54-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.554 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.554 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.554 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.554 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.554 DEBUG:boto:Method: GET 2023-09-29T22:22:41.554 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.554 DEBUG:boto:Data: 2023-09-29T22:22:41.555 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.555 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.555 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.555 DEBUG:boto:Params: {} 2023-09-29T22:22:41.555 DEBUG:boto:Token: None 2023-09-29T22:22:41.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.555 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw= 2023-09-29T22:22:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw=', 'Content-Length': '0'} 2023-09-29T22:22:41.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee3749bca6ba8c78-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.557 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.557 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.557 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.557 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.557 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.558 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.558 DEBUG:boto:Data: 2023-09-29T22:22:41.558 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.558 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.558 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.558 DEBUG:boto:Params: {} 2023-09-29T22:22:41.558 DEBUG:boto:Token: None 2023-09-29T22:22:41.558 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.558 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK/8dfidGe/QhfvWawhaNuNmKAs= 2023-09-29T22:22:41.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK/8dfidGe/QhfvWawhaNuNmKAs=', 'Content-Length': '0'} 2023-09-29T22:22:41.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015e041d314825814-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.560 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.561 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.561 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.561 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.561 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.561 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.561 DEBUG:boto:Data: 2023-09-29T22:22:41.561 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.561 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.561 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.561 DEBUG:boto:Params: {} 2023-09-29T22:22:41.562 DEBUG:boto:Token: None 2023-09-29T22:22:41.562 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.562 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tK/8dfidGe/QhfvWawhaNuNmKAs= 2023-09-29T22:22:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tK/8dfidGe/QhfvWawhaNuNmKAs=', 'Content-Length': '0'} 2023-09-29T22:22:41.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051be6bba922cedda-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.564 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.564 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.564 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.564 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.564 DEBUG:boto:Method: GET 2023-09-29T22:22:41.564 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.564 DEBUG:boto:Data: 2023-09-29T22:22:41.565 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.565 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.565 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.565 DEBUG:boto:Params: {} 2023-09-29T22:22:41.565 DEBUG:boto:Token: None 2023-09-29T22:22:41.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.565 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw= 2023-09-29T22:22:41.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw=', 'Content-Length': '0'} 2023-09-29T22:22:41.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9662f88c1e0587f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.567 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.567 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.567 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.567 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.567 DEBUG:boto:Method: GET 2023-09-29T22:22:41.568 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.568 DEBUG:boto:Data: 2023-09-29T22:22:41.568 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.568 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.568 DEBUG:boto:Params: {} 2023-09-29T22:22:41.568 DEBUG:boto:Token: None 2023-09-29T22:22:41.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:41.568 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw= 2023-09-29T22:22:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0Cbn1UQ2HLFX2eX4pxB+0EN37gw=', 'Content-Length': '0'} 2023-09-29T22:22:41.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c821846e80783136-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.570 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.570 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.570 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.570 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.571 DEBUG:boto:Method: GET 2023-09-29T22:22:41.571 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.571 DEBUG:boto:Data: 2023-09-29T22:22:41.571 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.571 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.571 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.571 DEBUG:boto:Params: {} 2023-09-29T22:22:41.571 DEBUG:boto:Token: None 2023-09-29T22:22:41.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84a8a742a691186-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.574 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.574 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.575 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.575 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.575 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.575 DEBUG:boto:Method: GET 2023-09-29T22:22:41.575 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.575 DEBUG:boto:Data: 2023-09-29T22:22:41.575 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.575 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.575 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.575 DEBUG:boto:Params: {} 2023-09-29T22:22:41.576 DEBUG:boto:Token: None 2023-09-29T22:22:41.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.576 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073459b447094adc8-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.578 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.578 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.578 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.578 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.578 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.579 DEBUG:boto:Data: 2023-09-29T22:22:41.579 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.579 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.579 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.579 DEBUG:boto:Params: {} 2023-09-29T22:22:41.579 DEBUG:boto:Token: None 2023-09-29T22:22:41.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab1c9de09b124119-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.582 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.583 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.583 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.583 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.583 DEBUG:boto:Data: 2023-09-29T22:22:41.583 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.583 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.583 DEBUG:boto:Params: {} 2023-09-29T22:22:41.583 DEBUG:boto:Token: None 2023-09-29T22:22:41.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000232079114557593d-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.587 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.587 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.587 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.587 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.587 DEBUG:boto:Method: GET 2023-09-29T22:22:41.587 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.587 DEBUG:boto:Data: 2023-09-29T22:22:41.587 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.587 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.588 DEBUG:boto:Params: {} 2023-09-29T22:22:41.588 DEBUG:boto:Token: None 2023-09-29T22:22:41.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e8c0cd51f0d679c-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.630 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.630 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.631 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.631 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.631 DEBUG:boto:Method: GET 2023-09-29T22:22:41.631 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.631 DEBUG:boto:Data: 2023-09-29T22:22:41.631 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.631 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.631 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.631 DEBUG:boto:Params: {} 2023-09-29T22:22:41.631 DEBUG:boto:Token: None 2023-09-29T22:22:41.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.632 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000351e950581199ddf-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.633 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.634 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.634 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.634 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.634 DEBUG:boto:Method: GET 2023-09-29T22:22:41.634 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.634 DEBUG:boto:Data: 2023-09-29T22:22:41.634 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.634 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.634 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.634 DEBUG:boto:Params: {} 2023-09-29T22:22:41.635 DEBUG:boto:Token: None 2023-09-29T22:22:41.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.635 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03f19ff5c8e5664-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.638 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.638 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.638 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.638 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.638 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.638 DEBUG:boto:Method: GET 2023-09-29T22:22:41.638 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.638 DEBUG:boto:Data: 2023-09-29T22:22:41.639 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.639 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.639 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.639 DEBUG:boto:Params: {} 2023-09-29T22:22:41.639 DEBUG:boto:Token: None 2023-09-29T22:22:41.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.639 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034aa6080e8a2a19b-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.641 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.642 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.642 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.642 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.642 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.642 DEBUG:boto:Method: GET 2023-09-29T22:22:41.642 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.643 DEBUG:boto:Data: 2023-09-29T22:22:41.643 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.643 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.643 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.643 DEBUG:boto:Params: {} 2023-09-29T22:22:41.643 DEBUG:boto:Token: None 2023-09-29T22:22:41.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.643 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E= 2023-09-29T22:22:41.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E=', 'Content-Length': '0'} 2023-09-29T22:22:41.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a230e3d9e6d4b63-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.645 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.645 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.646 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.646 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.646 DEBUG:boto:Method: GET 2023-09-29T22:22:41.646 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.646 DEBUG:boto:Data: 2023-09-29T22:22:41.646 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.646 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.646 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.646 DEBUG:boto:Params: {} 2023-09-29T22:22:41.646 DEBUG:boto:Token: None 2023-09-29T22:22:41.647 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.647 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E= 2023-09-29T22:22:41.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E=', 'Content-Length': '0'} 2023-09-29T22:22:41.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084230cdca1b8e051-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.648 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.648 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.649 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.649 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.649 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.649 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.649 DEBUG:boto:Data: 2023-09-29T22:22:41.649 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.649 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.649 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.649 DEBUG:boto:Params: {} 2023-09-29T22:22:41.649 DEBUG:boto:Token: None 2023-09-29T22:22:41.650 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.650 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pUQ21opEJTaD3pPnuLF043/oKzs= 2023-09-29T22:22:41.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pUQ21opEJTaD3pPnuLF043/oKzs=', 'Content-Length': '0'} 2023-09-29T22:22:41.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8f122e49a927290-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.652 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.652 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.652 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.652 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.652 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.652 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.652 DEBUG:boto:Data: 2023-09-29T22:22:41.652 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.652 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.652 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.653 DEBUG:boto:Params: {} 2023-09-29T22:22:41.653 DEBUG:boto:Token: None 2023-09-29T22:22:41.653 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.653 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pUQ21opEJTaD3pPnuLF043/oKzs= 2023-09-29T22:22:41.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pUQ21opEJTaD3pPnuLF043/oKzs=', 'Content-Length': '0'} 2023-09-29T22:22:41.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000010a101273221dea-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.655 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.655 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.655 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.655 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.655 DEBUG:boto:Method: GET 2023-09-29T22:22:41.655 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.655 DEBUG:boto:Data: 2023-09-29T22:22:41.656 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.656 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.656 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.656 DEBUG:boto:Params: {} 2023-09-29T22:22:41.656 DEBUG:boto:Token: None 2023-09-29T22:22:41.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E= 2023-09-29T22:22:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E=', 'Content-Length': '0'} 2023-09-29T22:22:41.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000605e13d8ff5b72d1-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.659 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.659 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.659 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.659 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.659 DEBUG:boto:Method: GET 2023-09-29T22:22:41.659 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.659 DEBUG:boto:Data: 2023-09-29T22:22:41.659 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.659 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.660 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.660 DEBUG:boto:Params: {} 2023-09-29T22:22:41.660 DEBUG:boto:Token: None 2023-09-29T22:22:41.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:41.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E= 2023-09-29T22:22:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E2zJmt5qKNRy2meCdNR1VgEki7E=', 'Content-Length': '0'} 2023-09-29T22:22:41.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f90fe0a21e8de815-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.702 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.703 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.703 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.703 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.703 DEBUG:boto:Method: GET 2023-09-29T22:22:41.703 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.703 DEBUG:boto:Data: 2023-09-29T22:22:41.703 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.703 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.703 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.704 DEBUG:boto:Params: {} 2023-09-29T22:22:41.704 DEBUG:boto:Token: None 2023-09-29T22:22:41.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab77958042147411-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.707 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.707 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.707 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.707 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.707 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.707 DEBUG:boto:Method: GET 2023-09-29T22:22:41.708 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.708 DEBUG:boto:Data: 2023-09-29T22:22:41.708 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.708 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.708 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.708 DEBUG:boto:Params: {} 2023-09-29T22:22:41.708 DEBUG:boto:Token: None 2023-09-29T22:22:41.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.708 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ec58b2ffb5805d-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.711 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.711 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.711 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.712 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.712 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.712 DEBUG:boto:Data: 2023-09-29T22:22:41.712 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.712 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.712 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.712 DEBUG:boto:Params: {} 2023-09-29T22:22:41.712 DEBUG:boto:Token: None 2023-09-29T22:22:41.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.713 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000be6fad9a3221d3a-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.716 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.716 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.716 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.716 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.716 DEBUG:boto:Data: 2023-09-29T22:22:41.716 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.716 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.716 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.717 DEBUG:boto:Params: {} 2023-09-29T22:22:41.717 DEBUG:boto:Token: None 2023-09-29T22:22:41.717 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.717 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b0d070fdc628e34-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.720 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.720 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.720 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.720 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.721 DEBUG:boto:Method: GET 2023-09-29T22:22:41.721 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.721 DEBUG:boto:Data: 2023-09-29T22:22:41.721 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.721 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.721 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.721 DEBUG:boto:Params: {} 2023-09-29T22:22:41.721 DEBUG:boto:Token: None 2023-09-29T22:22:41.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.722 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c698df9fa2bc01f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.766 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.766 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.767 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.767 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.767 DEBUG:boto:Method: GET 2023-09-29T22:22:41.767 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.767 DEBUG:boto:Data: 2023-09-29T22:22:41.767 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.767 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.767 DEBUG:boto:Params: {} 2023-09-29T22:22:41.767 DEBUG:boto:Token: None 2023-09-29T22:22:41.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000700b4398a6da4ec0-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.770 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.770 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.770 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.770 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.770 DEBUG:boto:Method: GET 2023-09-29T22:22:41.770 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.770 DEBUG:boto:Data: 2023-09-29T22:22:41.770 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.771 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.771 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.771 DEBUG:boto:Params: {} 2023-09-29T22:22:41.771 DEBUG:boto:Token: None 2023-09-29T22:22:41.771 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723e9a500f11488d-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.774 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.774 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.774 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.774 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.774 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.774 DEBUG:boto:Method: GET 2023-09-29T22:22:41.775 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.775 DEBUG:boto:Data: 2023-09-29T22:22:41.775 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.775 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.775 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.775 DEBUG:boto:Params: {} 2023-09-29T22:22:41.775 DEBUG:boto:Token: None 2023-09-29T22:22:41.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.775 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca80e7417f9cbda3-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.778 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.778 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.778 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.779 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.779 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.779 DEBUG:boto:Method: GET 2023-09-29T22:22:41.779 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.779 DEBUG:boto:Data: 2023-09-29T22:22:41.779 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.779 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.779 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.779 DEBUG:boto:Params: {} 2023-09-29T22:22:41.779 DEBUG:boto:Token: None 2023-09-29T22:22:41.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA= 2023-09-29T22:22:41.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020c7a1bba178579e-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.782 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.782 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.782 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.782 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.782 DEBUG:boto:Method: GET 2023-09-29T22:22:41.782 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.782 DEBUG:boto:Data: 2023-09-29T22:22:41.782 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.783 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.783 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.783 DEBUG:boto:Params: {} 2023-09-29T22:22:41.783 DEBUG:boto:Token: None 2023-09-29T22:22:41.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.783 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA= 2023-09-29T22:22:41.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b578119f65bc7def-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.785 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.785 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.785 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.785 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.785 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.785 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.785 DEBUG:boto:Data: 2023-09-29T22:22:41.785 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.786 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.786 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.786 DEBUG:boto:Params: {} 2023-09-29T22:22:41.786 DEBUG:boto:Token: None 2023-09-29T22:22:41.786 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.786 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:flP/Ou9FE35hT7DCIMF8keTkKWQ= 2023-09-29T22:22:41.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:flP/Ou9FE35hT7DCIMF8keTkKWQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000032752090858967c-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.788 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.788 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.788 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.789 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.789 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.789 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.789 DEBUG:boto:Data: 2023-09-29T22:22:41.789 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.789 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.789 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.789 DEBUG:boto:Params: {} 2023-09-29T22:22:41.789 DEBUG:boto:Token: None 2023-09-29T22:22:41.790 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.790 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:flP/Ou9FE35hT7DCIMF8keTkKWQ= 2023-09-29T22:22:41.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:flP/Ou9FE35hT7DCIMF8keTkKWQ=', 'Content-Length': '0'} 2023-09-29T22:22:41.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9d9f1981fa35959-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.792 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.792 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.792 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.792 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.792 DEBUG:boto:Method: GET 2023-09-29T22:22:41.793 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.793 DEBUG:boto:Data: 2023-09-29T22:22:41.793 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.793 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.793 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.793 DEBUG:boto:Params: {} 2023-09-29T22:22:41.793 DEBUG:boto:Token: None 2023-09-29T22:22:41.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.793 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA= 2023-09-29T22:22:41.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008433327fe9a62a7f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.795 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.795 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.796 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.796 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.796 DEBUG:boto:Method: GET 2023-09-29T22:22:41.796 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.796 DEBUG:boto:Data: 2023-09-29T22:22:41.796 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.796 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.796 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.796 DEBUG:boto:Params: {} 2023-09-29T22:22:41.796 DEBUG:boto:Token: None 2023-09-29T22:22:41.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:41.797 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA= 2023-09-29T22:22:41.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwBzYfQwq1Q3yhsITNROjGdpkgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ca659ec681b697e-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.799 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.799 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.799 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.799 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.799 DEBUG:boto:Method: GET 2023-09-29T22:22:41.799 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.799 DEBUG:boto:Data: 2023-09-29T22:22:41.799 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.800 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.800 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.800 DEBUG:boto:Params: {} 2023-09-29T22:22:41.800 DEBUG:boto:Token: None 2023-09-29T22:22:41.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d901a8b913d4df8e-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.803 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.803 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.803 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.803 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.803 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.804 DEBUG:boto:Method: GET 2023-09-29T22:22:41.804 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.804 DEBUG:boto:Data: 2023-09-29T22:22:41.804 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.804 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.804 DEBUG:boto:Params: {} 2023-09-29T22:22:41.804 DEBUG:boto:Token: None 2023-09-29T22:22:41.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007835abd9b675fe90-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.850 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.851 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.851 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.851 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.851 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.851 DEBUG:boto:Data: 2023-09-29T22:22:41.851 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.852 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.852 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.852 DEBUG:boto:Params: {} 2023-09-29T22:22:41.852 DEBUG:boto:Token: None 2023-09-29T22:22:41.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3e31f03f8896cfc-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.856 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.856 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.856 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.856 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.856 DEBUG:boto:Data: 2023-09-29T22:22:41.856 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.856 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.856 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.856 DEBUG:boto:Params: {} 2023-09-29T22:22:41.857 DEBUG:boto:Token: None 2023-09-29T22:22:41.857 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.857 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2ac59a237b282e0-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.860 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.860 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.860 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.861 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.861 DEBUG:boto:Method: GET 2023-09-29T22:22:41.861 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.861 DEBUG:boto:Data: 2023-09-29T22:22:41.861 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.861 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.861 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.861 DEBUG:boto:Params: {} 2023-09-29T22:22:41.861 DEBUG:boto:Token: None 2023-09-29T22:22:41.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.862 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa5c895d990d4976-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.906 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.906 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.907 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.907 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.907 DEBUG:boto:Method: GET 2023-09-29T22:22:41.907 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.907 DEBUG:boto:Data: 2023-09-29T22:22:41.907 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.907 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.907 DEBUG:boto:Params: {} 2023-09-29T22:22:41.907 DEBUG:boto:Token: None 2023-09-29T22:22:41.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e644ce3a456f0918-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.910 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.910 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.910 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.910 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.910 DEBUG:boto:Method: GET 2023-09-29T22:22:41.910 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.910 DEBUG:boto:Data: 2023-09-29T22:22:41.911 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.911 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.911 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.911 DEBUG:boto:Params: {} 2023-09-29T22:22:41.911 DEBUG:boto:Token: None 2023-09-29T22:22:41.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.911 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d069da186936fad9-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.914 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.914 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.914 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.914 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.914 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.915 DEBUG:boto:Method: GET 2023-09-29T22:22:41.915 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.915 DEBUG:boto:Data: 2023-09-29T22:22:41.915 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.915 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.915 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.915 DEBUG:boto:Params: {} 2023-09-29T22:22:41.915 DEBUG:boto:Token: None 2023-09-29T22:22:41.915 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.915 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006179b1dd73332d1b-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.918 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.918 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.919 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.919 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.919 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.919 DEBUG:boto:Method: GET 2023-09-29T22:22:41.919 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.919 DEBUG:boto:Data: 2023-09-29T22:22:41.919 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.919 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.919 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.919 DEBUG:boto:Params: {} 2023-09-29T22:22:41.920 DEBUG:boto:Token: None 2023-09-29T22:22:41.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o= 2023-09-29T22:22:41.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o=', 'Content-Length': '0'} 2023-09-29T22:22:41.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076e5789f4b103ece-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.922 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.923 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.923 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.923 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.923 DEBUG:boto:Method: GET 2023-09-29T22:22:41.923 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.923 DEBUG:boto:Data: 2023-09-29T22:22:41.923 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.923 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.923 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.923 DEBUG:boto:Params: {} 2023-09-29T22:22:41.924 DEBUG:boto:Token: None 2023-09-29T22:22:41.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o= 2023-09-29T22:22:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o=', 'Content-Length': '0'} 2023-09-29T22:22:41.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ce6f4fbaa20348d-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.926 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.926 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.926 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.926 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.926 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.926 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.926 DEBUG:boto:Data: 2023-09-29T22:22:41.927 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.927 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.927 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.927 DEBUG:boto:Params: {} 2023-09-29T22:22:41.927 DEBUG:boto:Token: None 2023-09-29T22:22:41.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.927 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WTOUxwlCXLVzv7zTIk2y5KuxqDk= 2023-09-29T22:22:41.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WTOUxwlCXLVzv7zTIk2y5KuxqDk=', 'Content-Length': '0'} 2023-09-29T22:22:41.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000693c28afb437456a-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.929 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.929 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.929 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.929 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.929 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.930 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.930 DEBUG:boto:Data: 2023-09-29T22:22:41.930 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.930 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.930 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.930 DEBUG:boto:Params: {} 2023-09-29T22:22:41.930 DEBUG:boto:Token: None 2023-09-29T22:22:41.930 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.930 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WTOUxwlCXLVzv7zTIk2y5KuxqDk= 2023-09-29T22:22:41.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WTOUxwlCXLVzv7zTIk2y5KuxqDk=', 'Content-Length': '0'} 2023-09-29T22:22:41.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fff7074ee6296286-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.932 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.933 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.933 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.933 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.933 DEBUG:boto:Method: GET 2023-09-29T22:22:41.933 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.933 DEBUG:boto:Data: 2023-09-29T22:22:41.933 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.933 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.933 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.933 DEBUG:boto:Params: {} 2023-09-29T22:22:41.933 DEBUG:boto:Token: None 2023-09-29T22:22:41.934 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.934 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o= 2023-09-29T22:22:41.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o=', 'Content-Length': '0'} 2023-09-29T22:22:41.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001be2d8caa482e9ef-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.936 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.936 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.936 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.936 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.936 DEBUG:boto:Method: GET 2023-09-29T22:22:41.936 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.936 DEBUG:boto:Data: 2023-09-29T22:22:41.936 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.937 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.937 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.937 DEBUG:boto:Params: {} 2023-09-29T22:22:41.937 DEBUG:boto:Token: None 2023-09-29T22:22:41.937 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:41.937 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o= 2023-09-29T22:22:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4MwJ40nlG4rJz/Yr0vz1Slxk54o=', 'Content-Length': '0'} 2023-09-29T22:22:41.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cc89a0038cb874a-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.939 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.939 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.939 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.940 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.940 DEBUG:boto:Method: GET 2023-09-29T22:22:41.940 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.940 DEBUG:boto:Data: 2023-09-29T22:22:41.940 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.940 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.940 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.940 DEBUG:boto:Params: {} 2023-09-29T22:22:41.940 DEBUG:boto:Token: None 2023-09-29T22:22:41.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.941 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a64e68903237dcb-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.943 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.944 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:41.944 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:41.944 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.944 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.944 DEBUG:boto:Method: GET 2023-09-29T22:22:41.944 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:41.944 DEBUG:boto:Data: 2023-09-29T22:22:41.944 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.944 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.945 DEBUG:boto:Params: {} 2023-09-29T22:22:41.945 DEBUG:boto:Token: None 2023-09-29T22:22:41.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/ 2023-09-29T22:22:41.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM= 2023-09-29T22:22:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/7Q89yhWCJ09JcksKahpEqZutvM=', 'Content-Length': '0'} 2023-09-29T22:22:41.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0cbb70ff994fac-0065174e31-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.947 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:41.948 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.948 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.948 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.948 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.948 DEBUG:boto:Data: 2023-09-29T22:22:41.948 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.948 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.948 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.948 DEBUG:boto:Params: {} 2023-09-29T22:22:41.949 DEBUG:boto:Token: None 2023-09-29T22:22:41.949 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000248c6a2deaeca26f-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.952 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.952 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.952 DEBUG:boto:Method: HEAD 2023-09-29T22:22:41.953 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:41.953 DEBUG:boto:Data: 2023-09-29T22:22:41.953 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.953 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:41.953 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.953 DEBUG:boto:Params: {} 2023-09-29T22:22:41.953 DEBUG:boto:Token: None 2023-09-29T22:22:41.953 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj 2023-09-29T22:22:41.953 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA= 2023-09-29T22:22:41.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M04DiGKP6XeqiSmHovMmZtiWxgA=', 'Content-Length': '0'} 2023-09-29T22:22:41.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af1063b6fdc638a8-0065174e31-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:41.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:41.957 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:41.957 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:41.957 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.957 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.957 DEBUG:boto:Method: GET 2023-09-29T22:22:41.957 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.957 DEBUG:boto:Data: 2023-09-29T22:22:41.957 DEBUG:boto:Headers: {} 2023-09-29T22:22:41.958 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:41.958 DEBUG:boto:Port: 8000 2023-09-29T22:22:41.958 DEBUG:boto:Params: {} 2023-09-29T22:22:41.958 DEBUG:boto:Token: None 2023-09-29T22:22:41.958 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:41 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:41.958 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog= 2023-09-29T22:22:41.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:El77b5uKTv72YRcxrMTgGd5bXog=', 'Content-Length': '0'} 2023-09-29T22:22:41.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e309a08c8df40921-0065174e31-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.002 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.002 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.003 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.003 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.003 DEBUG:boto:Method: GET 2023-09-29T22:22:42.003 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.003 DEBUG:boto:Data: 2023-09-29T22:22:42.003 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.003 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.003 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.003 DEBUG:boto:Params: {} 2023-09-29T22:22:42.003 DEBUG:boto:Token: None 2023-09-29T22:22:42.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001551bcc97d3c0643-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.005 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.005 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.006 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.006 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.006 DEBUG:boto:Method: GET 2023-09-29T22:22:42.006 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.006 DEBUG:boto:Data: 2023-09-29T22:22:42.006 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.006 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.006 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.006 DEBUG:boto:Params: {} 2023-09-29T22:22:42.006 DEBUG:boto:Token: None 2023-09-29T22:22:42.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a599d062141b1961-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.009 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.010 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.010 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.010 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.010 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.010 DEBUG:boto:Method: GET 2023-09-29T22:22:42.010 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.010 DEBUG:boto:Data: 2023-09-29T22:22:42.010 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.011 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.011 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.011 DEBUG:boto:Params: {} 2023-09-29T22:22:42.011 DEBUG:boto:Token: None 2023-09-29T22:22:42.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e31435bafb64faf-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.014 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.014 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.014 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.014 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.014 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.014 DEBUG:boto:Method: GET 2023-09-29T22:22:42.015 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.015 DEBUG:boto:Data: 2023-09-29T22:22:42.015 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.015 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.015 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.015 DEBUG:boto:Params: {} 2023-09-29T22:22:42.015 DEBUG:boto:Token: None 2023-09-29T22:22:42.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc= 2023-09-29T22:22:42.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc=', 'Content-Length': '0'} 2023-09-29T22:22:42.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065eaa7e789c1b033-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.017 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.017 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.018 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.018 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.018 DEBUG:boto:Method: GET 2023-09-29T22:22:42.018 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.018 DEBUG:boto:Data: 2023-09-29T22:22:42.018 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.018 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.018 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.018 DEBUG:boto:Params: {} 2023-09-29T22:22:42.018 DEBUG:boto:Token: None 2023-09-29T22:22:42.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.019 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc= 2023-09-29T22:22:42.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc=', 'Content-Length': '0'} 2023-09-29T22:22:42.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb3831a07fa6dad5-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.021 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.021 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.021 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.021 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.021 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.021 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.021 DEBUG:boto:Data: 2023-09-29T22:22:42.021 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.021 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.021 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.022 DEBUG:boto:Params: {} 2023-09-29T22:22:42.022 DEBUG:boto:Token: None 2023-09-29T22:22:42.022 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.022 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+CilttDkUUsgMtwPvsC0qeJpjqs= 2023-09-29T22:22:42.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+CilttDkUUsgMtwPvsC0qeJpjqs=', 'Content-Length': '0'} 2023-09-29T22:22:42.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099131de787d683f9-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.024 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.024 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.024 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.024 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.024 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.025 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.025 DEBUG:boto:Data: 2023-09-29T22:22:42.025 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.025 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.025 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.025 DEBUG:boto:Params: {} 2023-09-29T22:22:42.025 DEBUG:boto:Token: None 2023-09-29T22:22:42.025 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+CilttDkUUsgMtwPvsC0qeJpjqs= 2023-09-29T22:22:42.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+CilttDkUUsgMtwPvsC0qeJpjqs=', 'Content-Length': '0'} 2023-09-29T22:22:42.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7c0edaeb767b933-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.027 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.027 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.027 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.028 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.028 DEBUG:boto:Method: GET 2023-09-29T22:22:42.028 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.028 DEBUG:boto:Data: 2023-09-29T22:22:42.028 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.028 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.028 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.028 DEBUG:boto:Params: {} 2023-09-29T22:22:42.028 DEBUG:boto:Token: None 2023-09-29T22:22:42.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc= 2023-09-29T22:22:42.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc=', 'Content-Length': '0'} 2023-09-29T22:22:42.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1945fb3a1eac17-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.031 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.031 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.031 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.031 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.031 DEBUG:boto:Method: GET 2023-09-29T22:22:42.031 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.031 DEBUG:boto:Data: 2023-09-29T22:22:42.031 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.031 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.032 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.032 DEBUG:boto:Params: {} 2023-09-29T22:22:42.032 DEBUG:boto:Token: None 2023-09-29T22:22:42.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:42.032 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc= 2023-09-29T22:22:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7W+n39YtFXicaDPVEkt9U7wGzqc=', 'Content-Length': '0'} 2023-09-29T22:22:42.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e20146e9fbc1c3b-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.034 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.034 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.034 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.034 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.034 DEBUG:boto:Method: GET 2023-09-29T22:22:42.034 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.035 DEBUG:boto:Data: 2023-09-29T22:22:42.035 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.035 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.035 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.035 DEBUG:boto:Params: {} 2023-09-29T22:22:42.035 DEBUG:boto:Token: None 2023-09-29T22:22:42.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.035 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050dec2275dbfaf77-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.038 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.038 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.038 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.038 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.038 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.039 DEBUG:boto:Method: GET 2023-09-29T22:22:42.039 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.039 DEBUG:boto:Data: 2023-09-29T22:22:42.039 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.039 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.039 DEBUG:boto:Params: {} 2023-09-29T22:22:42.039 DEBUG:boto:Token: None 2023-09-29T22:22:42.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.039 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa7c73da174c474-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.042 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.042 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.042 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.043 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.043 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.043 DEBUG:boto:Data: 2023-09-29T22:22:42.043 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.043 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.043 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.043 DEBUG:boto:Params: {} 2023-09-29T22:22:42.043 DEBUG:boto:Token: None 2023-09-29T22:22:42.043 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a9b55106da4603-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.047 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.047 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.047 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.047 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.047 DEBUG:boto:Data: 2023-09-29T22:22:42.047 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.047 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.047 DEBUG:boto:Params: {} 2023-09-29T22:22:42.048 DEBUG:boto:Token: None 2023-09-29T22:22:42.048 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098b906c353f46735-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.051 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.051 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.051 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.051 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.051 DEBUG:boto:Method: GET 2023-09-29T22:22:42.051 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.051 DEBUG:boto:Data: 2023-09-29T22:22:42.051 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.052 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.052 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.052 DEBUG:boto:Params: {} 2023-09-29T22:22:42.052 DEBUG:boto:Token: None 2023-09-29T22:22:42.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001798bd27faf27f6a-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.098 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.099 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.099 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.099 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.099 DEBUG:boto:Method: GET 2023-09-29T22:22:42.099 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.099 DEBUG:boto:Data: 2023-09-29T22:22:42.099 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.099 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.100 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.100 DEBUG:boto:Params: {} 2023-09-29T22:22:42.100 DEBUG:boto:Token: None 2023-09-29T22:22:42.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.100 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cecd15c2396df4b0-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.102 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.102 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.102 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.103 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.103 DEBUG:boto:Method: GET 2023-09-29T22:22:42.103 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.103 DEBUG:boto:Data: 2023-09-29T22:22:42.103 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.103 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.103 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.103 DEBUG:boto:Params: {} 2023-09-29T22:22:42.103 DEBUG:boto:Token: None 2023-09-29T22:22:42.103 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd27dc22936ba44-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.106 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.107 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.107 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.107 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.107 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.107 DEBUG:boto:Method: GET 2023-09-29T22:22:42.107 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.107 DEBUG:boto:Data: 2023-09-29T22:22:42.107 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.107 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.108 DEBUG:boto:Params: {} 2023-09-29T22:22:42.108 DEBUG:boto:Token: None 2023-09-29T22:22:42.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b89cdecda35e9c1a-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.110 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.111 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.111 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.111 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.111 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.111 DEBUG:boto:Method: GET 2023-09-29T22:22:42.111 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.111 DEBUG:boto:Data: 2023-09-29T22:22:42.112 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.112 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.112 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.112 DEBUG:boto:Params: {} 2023-09-29T22:22:42.112 DEBUG:boto:Token: None 2023-09-29T22:22:42.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q= 2023-09-29T22:22:42.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e62942f4a5343665-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.114 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.115 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.115 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.115 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.115 DEBUG:boto:Method: GET 2023-09-29T22:22:42.115 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.115 DEBUG:boto:Data: 2023-09-29T22:22:42.115 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.115 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.115 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.115 DEBUG:boto:Params: {} 2023-09-29T22:22:42.116 DEBUG:boto:Token: None 2023-09-29T22:22:42.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q= 2023-09-29T22:22:42.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4365c1048d58240-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.118 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.118 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.118 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.118 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.118 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.119 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.119 DEBUG:boto:Data: 2023-09-29T22:22:42.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.119 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.119 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.119 DEBUG:boto:Params: {} 2023-09-29T22:22:42.119 DEBUG:boto:Token: None 2023-09-29T22:22:42.119 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.119 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gFGOVbpLAai2R9J7x3CLF6pKBb8= 2023-09-29T22:22:42.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gFGOVbpLAai2R9J7x3CLF6pKBb8=', 'Content-Length': '0'} 2023-09-29T22:22:42.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098d3a11df8a154f9-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.121 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.122 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.122 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.122 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.122 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.122 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.122 DEBUG:boto:Data: 2023-09-29T22:22:42.122 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.122 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.122 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.122 DEBUG:boto:Params: {} 2023-09-29T22:22:42.123 DEBUG:boto:Token: None 2023-09-29T22:22:42.123 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.123 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gFGOVbpLAai2R9J7x3CLF6pKBb8= 2023-09-29T22:22:42.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gFGOVbpLAai2R9J7x3CLF6pKBb8=', 'Content-Length': '0'} 2023-09-29T22:22:42.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d6a52a4a9081b95-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.125 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.125 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.125 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.125 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.125 DEBUG:boto:Method: GET 2023-09-29T22:22:42.125 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.126 DEBUG:boto:Data: 2023-09-29T22:22:42.126 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.126 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.126 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.126 DEBUG:boto:Params: {} 2023-09-29T22:22:42.126 DEBUG:boto:Token: None 2023-09-29T22:22:42.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.126 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q= 2023-09-29T22:22:42.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c20c44c7dec14fb2-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.129 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.129 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.129 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.129 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.129 DEBUG:boto:Method: GET 2023-09-29T22:22:42.129 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.129 DEBUG:boto:Data: 2023-09-29T22:22:42.129 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.130 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.130 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.130 DEBUG:boto:Params: {} 2023-09-29T22:22:42.130 DEBUG:boto:Token: None 2023-09-29T22:22:42.130 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:42.130 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q= 2023-09-29T22:22:42.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YgVDBDlsJcB1jlhs9xmtD1X5J0Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d1e438a074fd577-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.132 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.132 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.132 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.132 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.132 DEBUG:boto:Method: GET 2023-09-29T22:22:42.133 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.133 DEBUG:boto:Data: 2023-09-29T22:22:42.133 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.133 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.133 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.133 DEBUG:boto:Params: {} 2023-09-29T22:22:42.133 DEBUG:boto:Token: None 2023-09-29T22:22:42.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.133 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e5fde7af6c37ec-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.136 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.136 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.137 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.137 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.137 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.137 DEBUG:boto:Method: GET 2023-09-29T22:22:42.137 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.137 DEBUG:boto:Data: 2023-09-29T22:22:42.137 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.137 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.137 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.137 DEBUG:boto:Params: {} 2023-09-29T22:22:42.138 DEBUG:boto:Token: None 2023-09-29T22:22:42.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c429a468032e17c3-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.140 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.140 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.141 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.141 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.141 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.141 DEBUG:boto:Data: 2023-09-29T22:22:42.141 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.141 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.141 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.141 DEBUG:boto:Params: {} 2023-09-29T22:22:42.141 DEBUG:boto:Token: None 2023-09-29T22:22:42.142 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.142 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7120360e2f30a22-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.145 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.145 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.145 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.145 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.146 DEBUG:boto:Data: 2023-09-29T22:22:42.146 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.146 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.146 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.146 DEBUG:boto:Params: {} 2023-09-29T22:22:42.146 DEBUG:boto:Token: None 2023-09-29T22:22:42.146 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.146 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000416099ab54903615-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.149 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.149 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.150 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.150 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.150 DEBUG:boto:Method: GET 2023-09-29T22:22:42.150 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.150 DEBUG:boto:Data: 2023-09-29T22:22:42.150 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.150 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.150 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.150 DEBUG:boto:Params: {} 2023-09-29T22:22:42.151 DEBUG:boto:Token: None 2023-09-29T22:22:42.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006da4b5117a460444-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.198 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.198 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.199 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.199 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.199 DEBUG:boto:Method: GET 2023-09-29T22:22:42.199 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.199 DEBUG:boto:Data: 2023-09-29T22:22:42.199 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.199 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.199 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.199 DEBUG:boto:Params: {} 2023-09-29T22:22:42.199 DEBUG:boto:Token: None 2023-09-29T22:22:42.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6fe43e3624087b0-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.202 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.202 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.202 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.202 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.202 DEBUG:boto:Method: GET 2023-09-29T22:22:42.202 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.202 DEBUG:boto:Data: 2023-09-29T22:22:42.202 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.203 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.203 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.203 DEBUG:boto:Params: {} 2023-09-29T22:22:42.203 DEBUG:boto:Token: None 2023-09-29T22:22:42.203 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.203 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000241481666c2541c9-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.206 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.206 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.206 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.206 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.207 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.207 DEBUG:boto:Method: GET 2023-09-29T22:22:42.207 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.207 DEBUG:boto:Data: 2023-09-29T22:22:42.207 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.207 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.207 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.207 DEBUG:boto:Params: {} 2023-09-29T22:22:42.207 DEBUG:boto:Token: None 2023-09-29T22:22:42.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d267a70fdc88d65-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.210 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.211 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.211 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.211 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.211 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.211 DEBUG:boto:Method: GET 2023-09-29T22:22:42.211 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.211 DEBUG:boto:Data: 2023-09-29T22:22:42.211 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.211 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.212 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.212 DEBUG:boto:Params: {} 2023-09-29T22:22:42.212 DEBUG:boto:Token: None 2023-09-29T22:22:42.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q= 2023-09-29T22:22:42.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088ec304670fc5fde-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.214 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.214 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.215 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.215 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.215 DEBUG:boto:Method: GET 2023-09-29T22:22:42.215 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.215 DEBUG:boto:Data: 2023-09-29T22:22:42.215 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.215 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.215 DEBUG:boto:Params: {} 2023-09-29T22:22:42.215 DEBUG:boto:Token: None 2023-09-29T22:22:42.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.216 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q= 2023-09-29T22:22:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e684caeb163ca784-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.218 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.218 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.218 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.218 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.219 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.219 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.219 DEBUG:boto:Data: 2023-09-29T22:22:42.219 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.219 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.219 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.219 DEBUG:boto:Params: {} 2023-09-29T22:22:42.219 DEBUG:boto:Token: None 2023-09-29T22:22:42.219 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HQ0o6vPNjSdTzqSVdRffQGeWKcA= 2023-09-29T22:22:42.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HQ0o6vPNjSdTzqSVdRffQGeWKcA=', 'Content-Length': '0'} 2023-09-29T22:22:42.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000095b4c7e6c892350-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.222 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.222 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.222 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.222 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.222 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.222 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.222 DEBUG:boto:Data: 2023-09-29T22:22:42.222 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.222 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.223 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.223 DEBUG:boto:Params: {} 2023-09-29T22:22:42.223 DEBUG:boto:Token: None 2023-09-29T22:22:42.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HQ0o6vPNjSdTzqSVdRffQGeWKcA= 2023-09-29T22:22:42.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HQ0o6vPNjSdTzqSVdRffQGeWKcA=', 'Content-Length': '0'} 2023-09-29T22:22:42.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6e94940bfef3f31-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.225 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.225 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.225 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.225 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.225 DEBUG:boto:Method: GET 2023-09-29T22:22:42.225 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.225 DEBUG:boto:Data: 2023-09-29T22:22:42.226 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.226 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.226 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.226 DEBUG:boto:Params: {} 2023-09-29T22:22:42.226 DEBUG:boto:Token: None 2023-09-29T22:22:42.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q= 2023-09-29T22:22:42.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c3d0b39a664adda-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.228 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.228 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.228 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.229 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.229 DEBUG:boto:Method: GET 2023-09-29T22:22:42.229 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.229 DEBUG:boto:Data: 2023-09-29T22:22:42.229 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.229 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.229 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.229 DEBUG:boto:Params: {} 2023-09-29T22:22:42.229 DEBUG:boto:Token: None 2023-09-29T22:22:42.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:42.230 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q= 2023-09-29T22:22:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pU1raBPgUzTASTvKDijjQCIJB2Q=', 'Content-Length': '0'} 2023-09-29T22:22:42.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087cb6497d546f11f-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.232 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.232 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.232 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.232 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.232 DEBUG:boto:Method: GET 2023-09-29T22:22:42.232 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.232 DEBUG:boto:Data: 2023-09-29T22:22:42.232 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.233 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.233 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.233 DEBUG:boto:Params: {} 2023-09-29T22:22:42.233 DEBUG:boto:Token: None 2023-09-29T22:22:42.233 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.233 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a85fab50a05470aa-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.236 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.236 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.236 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.236 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.236 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.237 DEBUG:boto:Method: GET 2023-09-29T22:22:42.237 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.237 DEBUG:boto:Data: 2023-09-29T22:22:42.237 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.237 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.237 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.237 DEBUG:boto:Params: {} 2023-09-29T22:22:42.237 DEBUG:boto:Token: None 2023-09-29T22:22:42.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc07f0758a5b043c-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.241 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.241 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.242 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.242 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.242 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.242 DEBUG:boto:Data: 2023-09-29T22:22:42.242 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.242 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.242 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.242 DEBUG:boto:Params: {} 2023-09-29T22:22:42.242 DEBUG:boto:Token: None 2023-09-29T22:22:42.243 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.243 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cebcf47073ca56e-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.246 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.246 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.246 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.246 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.246 DEBUG:boto:Data: 2023-09-29T22:22:42.247 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.247 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.247 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.247 DEBUG:boto:Params: {} 2023-09-29T22:22:42.247 DEBUG:boto:Token: None 2023-09-29T22:22:42.247 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.247 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002baec41b0edf30e8-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.250 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.250 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.250 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.251 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.251 DEBUG:boto:Method: GET 2023-09-29T22:22:42.251 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.251 DEBUG:boto:Data: 2023-09-29T22:22:42.251 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.251 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.251 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.251 DEBUG:boto:Params: {} 2023-09-29T22:22:42.251 DEBUG:boto:Token: None 2023-09-29T22:22:42.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007045e282371dbafb-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.294 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.294 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.295 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.295 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.295 DEBUG:boto:Method: GET 2023-09-29T22:22:42.295 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.295 DEBUG:boto:Data: 2023-09-29T22:22:42.295 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.295 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.295 DEBUG:boto:Params: {} 2023-09-29T22:22:42.295 DEBUG:boto:Token: None 2023-09-29T22:22:42.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c53647a3c85cba89-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.298 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.298 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.298 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.298 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.299 DEBUG:boto:Method: GET 2023-09-29T22:22:42.299 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.299 DEBUG:boto:Data: 2023-09-29T22:22:42.299 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.299 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.299 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.299 DEBUG:boto:Params: {} 2023-09-29T22:22:42.299 DEBUG:boto:Token: None 2023-09-29T22:22:42.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.300 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df507b5dbc6838b9-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.342 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.343 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.343 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.343 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.343 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.343 DEBUG:boto:Method: GET 2023-09-29T22:22:42.343 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.343 DEBUG:boto:Data: 2023-09-29T22:22:42.343 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.343 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.343 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.343 DEBUG:boto:Params: {} 2023-09-29T22:22:42.344 DEBUG:boto:Token: None 2023-09-29T22:22:42.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c85be01eee5ba8c-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.347 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.347 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.347 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.347 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.347 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.347 DEBUG:boto:Method: GET 2023-09-29T22:22:42.347 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.348 DEBUG:boto:Data: 2023-09-29T22:22:42.348 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.348 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.348 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.348 DEBUG:boto:Params: {} 2023-09-29T22:22:42.348 DEBUG:boto:Token: None 2023-09-29T22:22:42.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.348 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM= 2023-09-29T22:22:42.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM=', 'Content-Length': '0'} 2023-09-29T22:22:42.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4d512834e136b30-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.394 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.394 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.395 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.395 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.395 DEBUG:boto:Method: GET 2023-09-29T22:22:42.395 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.395 DEBUG:boto:Data: 2023-09-29T22:22:42.395 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.395 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.395 DEBUG:boto:Params: {} 2023-09-29T22:22:42.396 DEBUG:boto:Token: None 2023-09-29T22:22:42.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM= 2023-09-29T22:22:42.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM=', 'Content-Length': '0'} 2023-09-29T22:22:42.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022db8761e9bcadbb-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.398 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.398 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.398 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.398 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.399 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.399 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.399 DEBUG:boto:Data: 2023-09-29T22:22:42.399 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.399 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.399 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.399 DEBUG:boto:Params: {} 2023-09-29T22:22:42.399 DEBUG:boto:Token: None 2023-09-29T22:22:42.399 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fWPdczEaRZROW5l8IUeUXI00s4g= 2023-09-29T22:22:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fWPdczEaRZROW5l8IUeUXI00s4g=', 'Content-Length': '0'} 2023-09-29T22:22:42.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f8043429d7fd0dd-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.402 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.402 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.402 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.402 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.402 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.402 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.402 DEBUG:boto:Data: 2023-09-29T22:22:42.403 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.403 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.403 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.403 DEBUG:boto:Params: {} 2023-09-29T22:22:42.403 DEBUG:boto:Token: None 2023-09-29T22:22:42.403 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.403 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fWPdczEaRZROW5l8IUeUXI00s4g= 2023-09-29T22:22:42.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fWPdczEaRZROW5l8IUeUXI00s4g=', 'Content-Length': '0'} 2023-09-29T22:22:42.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052f6cb8f4582a1ad-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.405 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.406 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.406 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.406 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.406 DEBUG:boto:Method: GET 2023-09-29T22:22:42.406 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.406 DEBUG:boto:Data: 2023-09-29T22:22:42.406 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.406 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.406 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.407 DEBUG:boto:Params: {} 2023-09-29T22:22:42.407 DEBUG:boto:Token: None 2023-09-29T22:22:42.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM= 2023-09-29T22:22:42.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM=', 'Content-Length': '0'} 2023-09-29T22:22:42.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a1eb4a623517603-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.450 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.451 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.451 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.451 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.451 DEBUG:boto:Method: GET 2023-09-29T22:22:42.451 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.451 DEBUG:boto:Data: 2023-09-29T22:22:42.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.451 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.451 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.452 DEBUG:boto:Params: {} 2023-09-29T22:22:42.452 DEBUG:boto:Token: None 2023-09-29T22:22:42.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:42.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM= 2023-09-29T22:22:42.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Djs/rJMPOe7Z2OLNMe+YA6PYqWM=', 'Content-Length': '0'} 2023-09-29T22:22:42.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000499e56c64e2db957-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.454 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.454 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.454 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.454 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.454 DEBUG:boto:Method: GET 2023-09-29T22:22:42.455 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.455 DEBUG:boto:Data: 2023-09-29T22:22:42.455 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.455 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.455 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.455 DEBUG:boto:Params: {} 2023-09-29T22:22:42.455 DEBUG:boto:Token: None 2023-09-29T22:22:42.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.455 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037771cb47c1f65dd-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.498 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.499 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.499 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.499 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.499 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.499 DEBUG:boto:Method: GET 2023-09-29T22:22:42.499 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.499 DEBUG:boto:Data: 2023-09-29T22:22:42.499 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.500 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.500 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.500 DEBUG:boto:Params: {} 2023-09-29T22:22:42.500 DEBUG:boto:Token: None 2023-09-29T22:22:42.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db47ea51afb5231e-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.503 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.503 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.503 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.503 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.503 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.504 DEBUG:boto:Data: 2023-09-29T22:22:42.504 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.504 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.504 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.504 DEBUG:boto:Params: {} 2023-09-29T22:22:42.504 DEBUG:boto:Token: None 2023-09-29T22:22:42.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4b72290d3a2a569-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.508 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.508 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.508 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.508 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.508 DEBUG:boto:Data: 2023-09-29T22:22:42.508 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.508 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.508 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.508 DEBUG:boto:Params: {} 2023-09-29T22:22:42.509 DEBUG:boto:Token: None 2023-09-29T22:22:42.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084a365947438ae05-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.512 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.512 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.512 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.512 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.512 DEBUG:boto:Method: GET 2023-09-29T22:22:42.513 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.513 DEBUG:boto:Data: 2023-09-29T22:22:42.513 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.513 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.513 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.513 DEBUG:boto:Params: {} 2023-09-29T22:22:42.513 DEBUG:boto:Token: None 2023-09-29T22:22:42.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5aba6bcd9f4b7c3-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.558 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.559 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.559 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.559 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.559 DEBUG:boto:Method: GET 2023-09-29T22:22:42.559 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.559 DEBUG:boto:Data: 2023-09-29T22:22:42.559 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.560 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.560 DEBUG:boto:Params: {} 2023-09-29T22:22:42.560 DEBUG:boto:Token: None 2023-09-29T22:22:42.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000763893539fc01224-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.562 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.562 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.563 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.563 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.563 DEBUG:boto:Method: GET 2023-09-29T22:22:42.563 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.563 DEBUG:boto:Data: 2023-09-29T22:22:42.563 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.563 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.563 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.563 DEBUG:boto:Params: {} 2023-09-29T22:22:42.563 DEBUG:boto:Token: None 2023-09-29T22:22:42.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001bffe2c40904029-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.606 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.607 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.607 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.607 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.607 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.607 DEBUG:boto:Method: GET 2023-09-29T22:22:42.607 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.607 DEBUG:boto:Data: 2023-09-29T22:22:42.607 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.607 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.608 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.608 DEBUG:boto:Params: {} 2023-09-29T22:22:42.608 DEBUG:boto:Token: None 2023-09-29T22:22:42.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.608 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008739c2df256cc7b5-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.611 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.611 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.611 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.611 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.611 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.611 DEBUG:boto:Method: GET 2023-09-29T22:22:42.612 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.612 DEBUG:boto:Data: 2023-09-29T22:22:42.612 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.612 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.612 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.612 DEBUG:boto:Params: {} 2023-09-29T22:22:42.612 DEBUG:boto:Token: None 2023-09-29T22:22:42.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.612 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8= 2023-09-29T22:22:42.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8=', 'Content-Length': '0'} 2023-09-29T22:22:42.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a4743186ff147f6-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.614 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.615 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.615 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.615 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.615 DEBUG:boto:Method: GET 2023-09-29T22:22:42.615 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.615 DEBUG:boto:Data: 2023-09-29T22:22:42.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.615 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.615 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.615 DEBUG:boto:Params: {} 2023-09-29T22:22:42.616 DEBUG:boto:Token: None 2023-09-29T22:22:42.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8= 2023-09-29T22:22:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8=', 'Content-Length': '0'} 2023-09-29T22:22:42.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae72d57d308e9bf-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.618 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.618 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.618 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.618 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.618 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.618 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.618 DEBUG:boto:Data: 2023-09-29T22:22:42.618 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.619 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.619 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.619 DEBUG:boto:Params: {} 2023-09-29T22:22:42.619 DEBUG:boto:Token: None 2023-09-29T22:22:42.619 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.619 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LJePSnjEZmY/sxUY6xntJlHZZIA= 2023-09-29T22:22:42.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LJePSnjEZmY/sxUY6xntJlHZZIA=', 'Content-Length': '0'} 2023-09-29T22:22:42.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000d633ad5a5cb983-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.621 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.621 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.621 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.622 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.622 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.622 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.622 DEBUG:boto:Data: 2023-09-29T22:22:42.622 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.622 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.622 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.622 DEBUG:boto:Params: {} 2023-09-29T22:22:42.622 DEBUG:boto:Token: None 2023-09-29T22:22:42.622 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.623 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LJePSnjEZmY/sxUY6xntJlHZZIA= 2023-09-29T22:22:42.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LJePSnjEZmY/sxUY6xntJlHZZIA=', 'Content-Length': '0'} 2023-09-29T22:22:42.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cedaf8e5d8af99a-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.625 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.625 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.625 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.625 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.625 DEBUG:boto:Method: GET 2023-09-29T22:22:42.625 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.625 DEBUG:boto:Data: 2023-09-29T22:22:42.625 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.626 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.626 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.626 DEBUG:boto:Params: {} 2023-09-29T22:22:42.626 DEBUG:boto:Token: None 2023-09-29T22:22:42.626 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.626 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8= 2023-09-29T22:22:42.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8=', 'Content-Length': '0'} 2023-09-29T22:22:42.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086cd4a47045c2356-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.670 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.670 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.671 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.671 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.671 DEBUG:boto:Method: GET 2023-09-29T22:22:42.671 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.671 DEBUG:boto:Data: 2023-09-29T22:22:42.671 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.671 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.671 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.671 DEBUG:boto:Params: {} 2023-09-29T22:22:42.671 DEBUG:boto:Token: None 2023-09-29T22:22:42.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:42.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8= 2023-09-29T22:22:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bUG5vGiEIsC7sH+mpN8kTV3Q6E8=', 'Content-Length': '0'} 2023-09-29T22:22:42.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0a36f271121c83c-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.674 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.674 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.674 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.674 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.674 DEBUG:boto:Method: GET 2023-09-29T22:22:42.675 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.675 DEBUG:boto:Data: 2023-09-29T22:22:42.675 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.675 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.675 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.675 DEBUG:boto:Params: {} 2023-09-29T22:22:42.675 DEBUG:boto:Token: None 2023-09-29T22:22:42.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.675 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd2e1739e538c7b7-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.722 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.723 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.723 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.723 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.723 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.723 DEBUG:boto:Method: GET 2023-09-29T22:22:42.723 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.723 DEBUG:boto:Data: 2023-09-29T22:22:42.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.724 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.724 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.724 DEBUG:boto:Params: {} 2023-09-29T22:22:42.724 DEBUG:boto:Token: None 2023-09-29T22:22:42.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a88ab0a1b21909-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.727 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.727 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.727 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.727 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.728 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.728 DEBUG:boto:Data: 2023-09-29T22:22:42.728 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.728 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.728 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.728 DEBUG:boto:Params: {} 2023-09-29T22:22:42.728 DEBUG:boto:Token: None 2023-09-29T22:22:42.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000934b1721808d36cf-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.732 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.732 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.732 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.732 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.732 DEBUG:boto:Data: 2023-09-29T22:22:42.732 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.732 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.732 DEBUG:boto:Params: {} 2023-09-29T22:22:42.733 DEBUG:boto:Token: None 2023-09-29T22:22:42.733 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.733 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002befe8aaf76972cb-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.736 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.736 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.736 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.736 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.737 DEBUG:boto:Method: GET 2023-09-29T22:22:42.737 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.737 DEBUG:boto:Data: 2023-09-29T22:22:42.737 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.737 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.737 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.737 DEBUG:boto:Params: {} 2023-09-29T22:22:42.737 DEBUG:boto:Token: None 2023-09-29T22:22:42.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.738 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051f7f52c1423bd64-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.786 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.786 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.786 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.787 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.787 DEBUG:boto:Method: GET 2023-09-29T22:22:42.787 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.787 DEBUG:boto:Data: 2023-09-29T22:22:42.787 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.787 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.787 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.787 DEBUG:boto:Params: {} 2023-09-29T22:22:42.787 DEBUG:boto:Token: None 2023-09-29T22:22:42.787 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b33c12048c9847d6-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.790 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.790 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.790 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.790 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.790 DEBUG:boto:Method: GET 2023-09-29T22:22:42.791 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.791 DEBUG:boto:Data: 2023-09-29T22:22:42.791 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.791 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.791 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.791 DEBUG:boto:Params: {} 2023-09-29T22:22:42.791 DEBUG:boto:Token: None 2023-09-29T22:22:42.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.791 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d16e2249e57b3c7-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.834 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.835 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.835 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.835 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.835 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.835 DEBUG:boto:Method: GET 2023-09-29T22:22:42.835 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.835 DEBUG:boto:Data: 2023-09-29T22:22:42.835 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.835 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.835 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.836 DEBUG:boto:Params: {} 2023-09-29T22:22:42.836 DEBUG:boto:Token: None 2023-09-29T22:22:42.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001696e65dbf81cef-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.838 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.839 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.839 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.839 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.839 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.839 DEBUG:boto:Method: GET 2023-09-29T22:22:42.839 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.840 DEBUG:boto:Data: 2023-09-29T22:22:42.840 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.840 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.840 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.840 DEBUG:boto:Params: {} 2023-09-29T22:22:42.840 DEBUG:boto:Token: None 2023-09-29T22:22:42.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw= 2023-09-29T22:22:42.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw=', 'Content-Length': '0'} 2023-09-29T22:22:42.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e48a62ea60a21232-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.842 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.843 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.843 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.843 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.843 DEBUG:boto:Method: GET 2023-09-29T22:22:42.843 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.843 DEBUG:boto:Data: 2023-09-29T22:22:42.843 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.843 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.843 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.843 DEBUG:boto:Params: {} 2023-09-29T22:22:42.844 DEBUG:boto:Token: None 2023-09-29T22:22:42.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.844 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw= 2023-09-29T22:22:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw=', 'Content-Length': '0'} 2023-09-29T22:22:42.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f18a672024caf924-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.846 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.846 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.846 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.846 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.847 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.847 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.847 DEBUG:boto:Data: 2023-09-29T22:22:42.847 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.847 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.847 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.847 DEBUG:boto:Params: {} 2023-09-29T22:22:42.847 DEBUG:boto:Token: None 2023-09-29T22:22:42.847 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cmCdx28PuKz3ynqqybp7BZe1CMM= 2023-09-29T22:22:42.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cmCdx28PuKz3ynqqybp7BZe1CMM=', 'Content-Length': '0'} 2023-09-29T22:22:42.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd933882a27c2e83-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.849 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.850 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.850 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.850 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.850 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.850 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.850 DEBUG:boto:Data: 2023-09-29T22:22:42.850 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.850 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.850 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.851 DEBUG:boto:Params: {} 2023-09-29T22:22:42.851 DEBUG:boto:Token: None 2023-09-29T22:22:42.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.853 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cmCdx28PuKz3ynqqybp7BZe1CMM= 2023-09-29T22:22:42.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cmCdx28PuKz3ynqqybp7BZe1CMM=', 'Content-Length': '0'} 2023-09-29T22:22:42.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc9223b0ca85b95f-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.855 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.855 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.856 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.856 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.856 DEBUG:boto:Method: GET 2023-09-29T22:22:42.856 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.856 DEBUG:boto:Data: 2023-09-29T22:22:42.856 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.856 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.856 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.856 DEBUG:boto:Params: {} 2023-09-29T22:22:42.856 DEBUG:boto:Token: None 2023-09-29T22:22:42.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.857 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw= 2023-09-29T22:22:42.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw=', 'Content-Length': '0'} 2023-09-29T22:22:42.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099e33b3342fce0c7-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.906 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.906 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.907 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.907 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.907 DEBUG:boto:Method: GET 2023-09-29T22:22:42.907 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.907 DEBUG:boto:Data: 2023-09-29T22:22:42.907 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.907 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.907 DEBUG:boto:Params: {} 2023-09-29T22:22:42.907 DEBUG:boto:Token: None 2023-09-29T22:22:42.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:42.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw= 2023-09-29T22:22:42.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Pxvr7w4O/mFmOc2a14UTRTU3hpw=', 'Content-Length': '0'} 2023-09-29T22:22:42.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7056633eadf9413-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.910 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.911 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.911 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.911 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.911 DEBUG:boto:Method: GET 2023-09-29T22:22:42.911 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.911 DEBUG:boto:Data: 2023-09-29T22:22:42.911 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.911 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.911 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.911 DEBUG:boto:Params: {} 2023-09-29T22:22:42.912 DEBUG:boto:Token: None 2023-09-29T22:22:42.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.912 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6b791e620f7151-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.954 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.955 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:42.955 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:42.955 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.955 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.955 DEBUG:boto:Method: GET 2023-09-29T22:22:42.955 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:42.955 DEBUG:boto:Data: 2023-09-29T22:22:42.955 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.955 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.955 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.955 DEBUG:boto:Params: {} 2023-09-29T22:22:42.956 DEBUG:boto:Token: None 2023-09-29T22:22:42.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/ 2023-09-29T22:22:42.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg= 2023-09-29T22:22:42.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:heX7/kJF15SUljH+Pw89qeuXgPg=', 'Content-Length': '0'} 2023-09-29T22:22:42.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c324b7443654d8-0065174e32-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.959 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:42.959 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.959 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.959 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.959 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.959 DEBUG:boto:Data: 2023-09-29T22:22:42.959 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.960 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.960 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.960 DEBUG:boto:Params: {} 2023-09-29T22:22:42.960 DEBUG:boto:Token: None 2023-09-29T22:22:42.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ccf439bc3826c3c-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.963 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.963 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.964 DEBUG:boto:Method: HEAD 2023-09-29T22:22:42.964 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:42.964 DEBUG:boto:Data: 2023-09-29T22:22:42.964 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:42.964 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.964 DEBUG:boto:Params: {} 2023-09-29T22:22:42.964 DEBUG:boto:Token: None 2023-09-29T22:22:42.964 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj 2023-09-29T22:22:42.965 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M= 2023-09-29T22:22:42.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JNydJz+8Q858mtOEBZZnTnjc19M=', 'Content-Length': '0'} 2023-09-29T22:22:42.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a489b0360514ec6-0065174e32-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:42.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:42.968 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:42.968 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:42.968 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.968 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.968 DEBUG:boto:Method: GET 2023-09-29T22:22:42.968 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.969 DEBUG:boto:Data: 2023-09-29T22:22:42.969 DEBUG:boto:Headers: {} 2023-09-29T22:22:42.969 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:42.969 DEBUG:boto:Port: 8000 2023-09-29T22:22:42.969 DEBUG:boto:Params: {} 2023-09-29T22:22:42.969 DEBUG:boto:Token: None 2023-09-29T22:22:42.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:42 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:42.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374= 2023-09-29T22:22:42.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wDGMt2iudUy+ATMAogIVUinu374=', 'Content-Length': '0'} 2023-09-29T22:22:42.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000678ca6a2e7ef4693-0065174e32-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.014 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.014 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.015 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.015 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.015 DEBUG:boto:Method: GET 2023-09-29T22:22:43.015 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.015 DEBUG:boto:Data: 2023-09-29T22:22:43.015 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.015 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.015 DEBUG:boto:Params: {} 2023-09-29T22:22:43.015 DEBUG:boto:Token: None 2023-09-29T22:22:43.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016799d1bba8d16cc-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.018 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.018 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.018 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.018 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.018 DEBUG:boto:Method: GET 2023-09-29T22:22:43.018 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.018 DEBUG:boto:Data: 2023-09-29T22:22:43.019 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.019 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.019 DEBUG:boto:Params: {} 2023-09-29T22:22:43.019 DEBUG:boto:Token: None 2023-09-29T22:22:43.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.019 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef11db293e531c35-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.062 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.063 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.063 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.063 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.063 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.063 DEBUG:boto:Method: GET 2023-09-29T22:22:43.063 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.063 DEBUG:boto:Data: 2023-09-29T22:22:43.063 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.063 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.064 DEBUG:boto:Params: {} 2023-09-29T22:22:43.064 DEBUG:boto:Token: None 2023-09-29T22:22:43.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acec44e2e38ed12e-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.067 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.067 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.067 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.067 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.068 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.068 DEBUG:boto:Method: GET 2023-09-29T22:22:43.068 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.068 DEBUG:boto:Data: 2023-09-29T22:22:43.068 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.068 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.068 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.068 DEBUG:boto:Params: {} 2023-09-29T22:22:43.068 DEBUG:boto:Token: None 2023-09-29T22:22:43.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ= 2023-09-29T22:22:43.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ=', 'Content-Length': '0'} 2023-09-29T22:22:43.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c9380771d87922-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.071 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.071 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.071 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.071 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.071 DEBUG:boto:Method: GET 2023-09-29T22:22:43.071 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.071 DEBUG:boto:Data: 2023-09-29T22:22:43.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.072 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.072 DEBUG:boto:Params: {} 2023-09-29T22:22:43.072 DEBUG:boto:Token: None 2023-09-29T22:22:43.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ= 2023-09-29T22:22:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ=', 'Content-Length': '0'} 2023-09-29T22:22:43.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee50c7dd39a3a377-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.122 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.122 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.122 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.123 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.123 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.123 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.123 DEBUG:boto:Data: 2023-09-29T22:22:43.123 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.123 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.123 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.123 DEBUG:boto:Params: {} 2023-09-29T22:22:43.123 DEBUG:boto:Token: None 2023-09-29T22:22:43.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Ahr89HmgeNA/SCEtqmkzg+Y+AY= 2023-09-29T22:22:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Ahr89HmgeNA/SCEtqmkzg+Y+AY=', 'Content-Length': '0'} 2023-09-29T22:22:43.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9c813edd60bfcac-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.126 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.126 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.126 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.126 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.126 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.126 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.126 DEBUG:boto:Data: 2023-09-29T22:22:43.126 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.127 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.127 DEBUG:boto:Params: {} 2023-09-29T22:22:43.127 DEBUG:boto:Token: None 2023-09-29T22:22:43.127 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.127 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Ahr89HmgeNA/SCEtqmkzg+Y+AY= 2023-09-29T22:22:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Ahr89HmgeNA/SCEtqmkzg+Y+AY=', 'Content-Length': '0'} 2023-09-29T22:22:43.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad9b410ba12c6f16-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.130 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.130 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.130 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.130 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.130 DEBUG:boto:Method: GET 2023-09-29T22:22:43.131 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.131 DEBUG:boto:Data: 2023-09-29T22:22:43.131 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.131 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.131 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.131 DEBUG:boto:Params: {} 2023-09-29T22:22:43.131 DEBUG:boto:Token: None 2023-09-29T22:22:43.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.131 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ= 2023-09-29T22:22:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ=', 'Content-Length': '0'} 2023-09-29T22:22:43.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704a3c6ff26483cc-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.133 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.133 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.134 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.134 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.134 DEBUG:boto:Method: GET 2023-09-29T22:22:43.134 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.134 DEBUG:boto:Data: 2023-09-29T22:22:43.134 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.134 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.134 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.134 DEBUG:boto:Params: {} 2023-09-29T22:22:43.134 DEBUG:boto:Token: None 2023-09-29T22:22:43.135 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:43.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ= 2023-09-29T22:22:43.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ino+A+pKHYgPeWUQ4KJWKwRY2fQ=', 'Content-Length': '0'} 2023-09-29T22:22:43.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ed98bdf1d75dc57-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.137 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.137 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.137 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.138 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.138 DEBUG:boto:Method: GET 2023-09-29T22:22:43.138 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.138 DEBUG:boto:Data: 2023-09-29T22:22:43.138 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.138 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.138 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.138 DEBUG:boto:Params: {} 2023-09-29T22:22:43.138 DEBUG:boto:Token: None 2023-09-29T22:22:43.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.139 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16e402034b2b307-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.141 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.141 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.141 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.141 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.142 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.142 DEBUG:boto:Method: GET 2023-09-29T22:22:43.142 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.142 DEBUG:boto:Data: 2023-09-29T22:22:43.142 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.142 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.142 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.142 DEBUG:boto:Params: {} 2023-09-29T22:22:43.142 DEBUG:boto:Token: None 2023-09-29T22:22:43.142 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.143 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe2c4ac6233e0f7-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.146 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.146 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.146 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.146 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.146 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.146 DEBUG:boto:Data: 2023-09-29T22:22:43.146 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.146 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.147 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.147 DEBUG:boto:Params: {} 2023-09-29T22:22:43.147 DEBUG:boto:Token: None 2023-09-29T22:22:43.147 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.147 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003466a9932e74382f-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.150 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.150 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.151 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.151 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.151 DEBUG:boto:Data: 2023-09-29T22:22:43.151 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.151 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.151 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.151 DEBUG:boto:Params: {} 2023-09-29T22:22:43.151 DEBUG:boto:Token: None 2023-09-29T22:22:43.151 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.152 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f464a0a82373573-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.155 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.155 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.155 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.155 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.155 DEBUG:boto:Method: GET 2023-09-29T22:22:43.156 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.156 DEBUG:boto:Data: 2023-09-29T22:22:43.156 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.156 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.156 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.156 DEBUG:boto:Params: {} 2023-09-29T22:22:43.156 DEBUG:boto:Token: None 2023-09-29T22:22:43.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.156 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2755ac70362c65a-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.202 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.203 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.203 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.203 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.203 DEBUG:boto:Method: GET 2023-09-29T22:22:43.203 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.203 DEBUG:boto:Data: 2023-09-29T22:22:43.203 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.203 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.203 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.203 DEBUG:boto:Params: {} 2023-09-29T22:22:43.204 DEBUG:boto:Token: None 2023-09-29T22:22:43.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000958a4c737f1b98d1-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.206 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.206 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.206 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.206 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.207 DEBUG:boto:Method: GET 2023-09-29T22:22:43.207 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.207 DEBUG:boto:Data: 2023-09-29T22:22:43.207 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.207 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.207 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.207 DEBUG:boto:Params: {} 2023-09-29T22:22:43.207 DEBUG:boto:Token: None 2023-09-29T22:22:43.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ad7202277d632a-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.210 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.211 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.211 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.211 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.211 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.211 DEBUG:boto:Method: GET 2023-09-29T22:22:43.211 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.211 DEBUG:boto:Data: 2023-09-29T22:22:43.211 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.212 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.212 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.212 DEBUG:boto:Params: {} 2023-09-29T22:22:43.212 DEBUG:boto:Token: None 2023-09-29T22:22:43.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002630f93399ae0ddb-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.215 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.215 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.215 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.215 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.215 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.216 DEBUG:boto:Method: GET 2023-09-29T22:22:43.216 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.216 DEBUG:boto:Data: 2023-09-29T22:22:43.216 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.216 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.216 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.216 DEBUG:boto:Params: {} 2023-09-29T22:22:43.216 DEBUG:boto:Token: None 2023-09-29T22:22:43.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.216 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs= 2023-09-29T22:22:43.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs=', 'Content-Length': '0'} 2023-09-29T22:22:43.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e96496046fdb6c56-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.262 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.262 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.263 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.263 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.263 DEBUG:boto:Method: GET 2023-09-29T22:22:43.263 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.263 DEBUG:boto:Data: 2023-09-29T22:22:43.263 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.263 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.263 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.263 DEBUG:boto:Params: {} 2023-09-29T22:22:43.263 DEBUG:boto:Token: None 2023-09-29T22:22:43.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs= 2023-09-29T22:22:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs=', 'Content-Length': '0'} 2023-09-29T22:22:43.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc34069e4672c68-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.266 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.266 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.266 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.266 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.266 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.266 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.266 DEBUG:boto:Data: 2023-09-29T22:22:43.266 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.267 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.267 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.267 DEBUG:boto:Params: {} 2023-09-29T22:22:43.267 DEBUG:boto:Token: None 2023-09-29T22:22:43.267 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6TSdZqrzV/Hzovj9SeKzEL6lX9g= 2023-09-29T22:22:43.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6TSdZqrzV/Hzovj9SeKzEL6lX9g=', 'Content-Length': '0'} 2023-09-29T22:22:43.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab6ede81c9eb52f7-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.269 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.269 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.269 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.269 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.269 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.270 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.270 DEBUG:boto:Data: 2023-09-29T22:22:43.270 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.270 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.270 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.270 DEBUG:boto:Params: {} 2023-09-29T22:22:43.270 DEBUG:boto:Token: None 2023-09-29T22:22:43.270 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6TSdZqrzV/Hzovj9SeKzEL6lX9g= 2023-09-29T22:22:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6TSdZqrzV/Hzovj9SeKzEL6lX9g=', 'Content-Length': '0'} 2023-09-29T22:22:43.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b93b67fa71640919-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.273 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.273 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.273 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.273 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.273 DEBUG:boto:Method: GET 2023-09-29T22:22:43.273 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.273 DEBUG:boto:Data: 2023-09-29T22:22:43.273 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.273 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.274 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.274 DEBUG:boto:Params: {} 2023-09-29T22:22:43.274 DEBUG:boto:Token: None 2023-09-29T22:22:43.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.274 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs= 2023-09-29T22:22:43.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs=', 'Content-Length': '0'} 2023-09-29T22:22:43.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d8722974fc4384a-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.318 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.318 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.318 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.319 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.319 DEBUG:boto:Method: GET 2023-09-29T22:22:43.319 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.319 DEBUG:boto:Data: 2023-09-29T22:22:43.319 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.319 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.319 DEBUG:boto:Params: {} 2023-09-29T22:22:43.319 DEBUG:boto:Token: None 2023-09-29T22:22:43.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:43.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs= 2023-09-29T22:22:43.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onbdyI755RLuq1iQwL/ZE4PrCzs=', 'Content-Length': '0'} 2023-09-29T22:22:43.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbec0dcddd4fd99e-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.322 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.322 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.322 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.322 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.322 DEBUG:boto:Method: GET 2023-09-29T22:22:43.323 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.323 DEBUG:boto:Data: 2023-09-29T22:22:43.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.323 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.323 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.323 DEBUG:boto:Params: {} 2023-09-29T22:22:43.323 DEBUG:boto:Token: None 2023-09-29T22:22:43.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16fa386caf6d1c0-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.326 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.326 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.326 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.326 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.326 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.327 DEBUG:boto:Method: GET 2023-09-29T22:22:43.327 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.327 DEBUG:boto:Data: 2023-09-29T22:22:43.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.327 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.327 DEBUG:boto:Params: {} 2023-09-29T22:22:43.327 DEBUG:boto:Token: None 2023-09-29T22:22:43.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba3829731174a071-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.330 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.330 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.330 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.331 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.331 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.331 DEBUG:boto:Data: 2023-09-29T22:22:43.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.331 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.331 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.331 DEBUG:boto:Params: {} 2023-09-29T22:22:43.331 DEBUG:boto:Token: None 2023-09-29T22:22:43.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e776f5bac82536f7-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.335 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.335 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.335 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.335 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.335 DEBUG:boto:Data: 2023-09-29T22:22:43.335 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.335 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.336 DEBUG:boto:Params: {} 2023-09-29T22:22:43.336 DEBUG:boto:Token: None 2023-09-29T22:22:43.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a665e38dbce7e745-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.339 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.339 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.339 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.339 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.340 DEBUG:boto:Method: GET 2023-09-29T22:22:43.340 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.340 DEBUG:boto:Data: 2023-09-29T22:22:43.340 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.340 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.340 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.340 DEBUG:boto:Params: {} 2023-09-29T22:22:43.340 DEBUG:boto:Token: None 2023-09-29T22:22:43.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.341 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ad9351c092a656-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.386 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.386 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.387 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.387 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.387 DEBUG:boto:Method: GET 2023-09-29T22:22:43.387 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.387 DEBUG:boto:Data: 2023-09-29T22:22:43.387 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.387 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.387 DEBUG:boto:Params: {} 2023-09-29T22:22:43.387 DEBUG:boto:Token: None 2023-09-29T22:22:43.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6ea9ed64f1c2442-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.390 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.390 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.390 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.390 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.390 DEBUG:boto:Method: GET 2023-09-29T22:22:43.390 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.391 DEBUG:boto:Data: 2023-09-29T22:22:43.391 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.391 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.391 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.391 DEBUG:boto:Params: {} 2023-09-29T22:22:43.391 DEBUG:boto:Token: None 2023-09-29T22:22:43.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.391 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a4d68cbf7016e11-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.394 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.394 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.394 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.394 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.394 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.395 DEBUG:boto:Method: GET 2023-09-29T22:22:43.395 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.395 DEBUG:boto:Data: 2023-09-29T22:22:43.395 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.395 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.395 DEBUG:boto:Params: {} 2023-09-29T22:22:43.395 DEBUG:boto:Token: None 2023-09-29T22:22:43.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac99c7ce15b4e9c7-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.398 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.399 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.399 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.399 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.399 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.399 DEBUG:boto:Method: GET 2023-09-29T22:22:43.399 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.399 DEBUG:boto:Data: 2023-09-29T22:22:43.399 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.399 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.400 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.400 DEBUG:boto:Params: {} 2023-09-29T22:22:43.400 DEBUG:boto:Token: None 2023-09-29T22:22:43.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k= 2023-09-29T22:22:43.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k=', 'Content-Length': '0'} 2023-09-29T22:22:43.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9768cc7c651d53d-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.402 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.402 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.402 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.402 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.403 DEBUG:boto:Method: GET 2023-09-29T22:22:43.403 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.403 DEBUG:boto:Data: 2023-09-29T22:22:43.403 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.403 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.403 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.403 DEBUG:boto:Params: {} 2023-09-29T22:22:43.403 DEBUG:boto:Token: None 2023-09-29T22:22:43.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.403 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k= 2023-09-29T22:22:43.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k=', 'Content-Length': '0'} 2023-09-29T22:22:43.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054f140a901d48993-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.406 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.406 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.406 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.406 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.406 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.406 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.406 DEBUG:boto:Data: 2023-09-29T22:22:43.407 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.407 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.407 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.407 DEBUG:boto:Params: {} 2023-09-29T22:22:43.407 DEBUG:boto:Token: None 2023-09-29T22:22:43.407 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ThmD5b2hXvqKPHT//nJ6chqxehU= 2023-09-29T22:22:43.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ThmD5b2hXvqKPHT//nJ6chqxehU=', 'Content-Length': '0'} 2023-09-29T22:22:43.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1df5ec9b8154662-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.409 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.409 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.409 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.409 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.410 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.410 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.410 DEBUG:boto:Data: 2023-09-29T22:22:43.410 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.410 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.410 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.410 DEBUG:boto:Params: {} 2023-09-29T22:22:43.410 DEBUG:boto:Token: None 2023-09-29T22:22:43.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.411 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ThmD5b2hXvqKPHT//nJ6chqxehU= 2023-09-29T22:22:43.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ThmD5b2hXvqKPHT//nJ6chqxehU=', 'Content-Length': '0'} 2023-09-29T22:22:43.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eecf2a5d0aad002-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.413 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.413 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.413 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.413 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.413 DEBUG:boto:Method: GET 2023-09-29T22:22:43.413 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.413 DEBUG:boto:Data: 2023-09-29T22:22:43.413 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.413 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.414 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.414 DEBUG:boto:Params: {} 2023-09-29T22:22:43.414 DEBUG:boto:Token: None 2023-09-29T22:22:43.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.414 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k= 2023-09-29T22:22:43.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k=', 'Content-Length': '0'} 2023-09-29T22:22:43.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fbdb08d9e2fd062-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.462 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.462 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.463 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.463 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.463 DEBUG:boto:Method: GET 2023-09-29T22:22:43.463 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.463 DEBUG:boto:Data: 2023-09-29T22:22:43.463 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.463 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.463 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.463 DEBUG:boto:Params: {} 2023-09-29T22:22:43.464 DEBUG:boto:Token: None 2023-09-29T22:22:43.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:43.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k= 2023-09-29T22:22:43.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hvtv5wFvUdKrPmN3/JNDAzse/7k=', 'Content-Length': '0'} 2023-09-29T22:22:43.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399cf8577a754e2f-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.466 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.466 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.466 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.466 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.466 DEBUG:boto:Method: GET 2023-09-29T22:22:43.466 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.466 DEBUG:boto:Data: 2023-09-29T22:22:43.466 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.467 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.467 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.467 DEBUG:boto:Params: {} 2023-09-29T22:22:43.467 DEBUG:boto:Token: None 2023-09-29T22:22:43.467 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.467 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650bf7d9554b5f9d-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.470 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.470 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.470 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.470 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.471 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.471 DEBUG:boto:Method: GET 2023-09-29T22:22:43.471 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.471 DEBUG:boto:Data: 2023-09-29T22:22:43.471 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.471 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.471 DEBUG:boto:Params: {} 2023-09-29T22:22:43.471 DEBUG:boto:Token: None 2023-09-29T22:22:43.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a67a096092386de-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.474 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.475 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.475 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.475 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.475 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.475 DEBUG:boto:Data: 2023-09-29T22:22:43.475 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.475 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.475 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.476 DEBUG:boto:Params: {} 2023-09-29T22:22:43.476 DEBUG:boto:Token: None 2023-09-29T22:22:43.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086300aa45f575496-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.479 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.479 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.479 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.479 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.479 DEBUG:boto:Data: 2023-09-29T22:22:43.479 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.480 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.480 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.480 DEBUG:boto:Params: {} 2023-09-29T22:22:43.480 DEBUG:boto:Token: None 2023-09-29T22:22:43.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015c30f289b24732d-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.484 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.484 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.484 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.484 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.484 DEBUG:boto:Method: GET 2023-09-29T22:22:43.484 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.484 DEBUG:boto:Data: 2023-09-29T22:22:43.484 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.484 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.485 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.485 DEBUG:boto:Params: {} 2023-09-29T22:22:43.485 DEBUG:boto:Token: None 2023-09-29T22:22:43.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.485 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3a06c98e3f4229b-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.530 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.531 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.531 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.531 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.531 DEBUG:boto:Method: GET 2023-09-29T22:22:43.531 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.531 DEBUG:boto:Data: 2023-09-29T22:22:43.531 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.531 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.531 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.531 DEBUG:boto:Params: {} 2023-09-29T22:22:43.532 DEBUG:boto:Token: None 2023-09-29T22:22:43.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035e9f72acf1d2e07-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.534 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.534 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.534 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.534 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.535 DEBUG:boto:Method: GET 2023-09-29T22:22:43.535 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.535 DEBUG:boto:Data: 2023-09-29T22:22:43.535 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.535 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.535 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.535 DEBUG:boto:Params: {} 2023-09-29T22:22:43.535 DEBUG:boto:Token: None 2023-09-29T22:22:43.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379dc4e94f5f2457-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.538 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.538 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.539 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.539 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.539 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.539 DEBUG:boto:Method: GET 2023-09-29T22:22:43.539 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.539 DEBUG:boto:Data: 2023-09-29T22:22:43.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.539 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.540 DEBUG:boto:Params: {} 2023-09-29T22:22:43.540 DEBUG:boto:Token: None 2023-09-29T22:22:43.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4464c96b5a0674-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.543 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.543 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.543 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.543 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.543 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.543 DEBUG:boto:Method: GET 2023-09-29T22:22:43.544 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.544 DEBUG:boto:Data: 2023-09-29T22:22:43.544 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.544 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.544 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.544 DEBUG:boto:Params: {} 2023-09-29T22:22:43.544 DEBUG:boto:Token: None 2023-09-29T22:22:43.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4= 2023-09-29T22:22:43.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4=', 'Content-Length': '0'} 2023-09-29T22:22:43.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4e03aae86dc0f41-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.590 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.590 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.591 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.591 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.591 DEBUG:boto:Method: GET 2023-09-29T22:22:43.591 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.591 DEBUG:boto:Data: 2023-09-29T22:22:43.591 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.591 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.591 DEBUG:boto:Params: {} 2023-09-29T22:22:43.591 DEBUG:boto:Token: None 2023-09-29T22:22:43.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4= 2023-09-29T22:22:43.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4=', 'Content-Length': '0'} 2023-09-29T22:22:43.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a4e8bef8c50c0d5-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.594 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.594 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.594 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.594 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.594 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.594 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.595 DEBUG:boto:Data: 2023-09-29T22:22:43.595 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.595 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.595 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.595 DEBUG:boto:Params: {} 2023-09-29T22:22:43.595 DEBUG:boto:Token: None 2023-09-29T22:22:43.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.595 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UPzHQ34SG2dwq81JL+TgjB1Q0eY= 2023-09-29T22:22:43.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UPzHQ34SG2dwq81JL+TgjB1Q0eY=', 'Content-Length': '0'} 2023-09-29T22:22:43.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098c0583e14c3a425-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.597 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.597 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.597 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.598 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.598 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.598 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.598 DEBUG:boto:Data: 2023-09-29T22:22:43.598 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.598 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.598 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.598 DEBUG:boto:Params: {} 2023-09-29T22:22:43.598 DEBUG:boto:Token: None 2023-09-29T22:22:43.598 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.599 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UPzHQ34SG2dwq81JL+TgjB1Q0eY= 2023-09-29T22:22:43.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UPzHQ34SG2dwq81JL+TgjB1Q0eY=', 'Content-Length': '0'} 2023-09-29T22:22:43.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a1e546771f5a872-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.601 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.601 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.601 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.601 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.601 DEBUG:boto:Method: GET 2023-09-29T22:22:43.601 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.601 DEBUG:boto:Data: 2023-09-29T22:22:43.601 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.602 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.602 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.602 DEBUG:boto:Params: {} 2023-09-29T22:22:43.602 DEBUG:boto:Token: None 2023-09-29T22:22:43.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4= 2023-09-29T22:22:43.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4=', 'Content-Length': '0'} 2023-09-29T22:22:43.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004207775ee83f2723-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.650 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.650 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.650 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.651 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.651 DEBUG:boto:Method: GET 2023-09-29T22:22:43.651 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.651 DEBUG:boto:Data: 2023-09-29T22:22:43.651 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.651 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.651 DEBUG:boto:Params: {} 2023-09-29T22:22:43.651 DEBUG:boto:Token: None 2023-09-29T22:22:43.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:43.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4= 2023-09-29T22:22:43.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IhgAu5DuvR+4OrylVe92PBICoR4=', 'Content-Length': '0'} 2023-09-29T22:22:43.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000304d3062761318d0-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.654 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.654 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.654 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.654 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.654 DEBUG:boto:Method: GET 2023-09-29T22:22:43.654 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.655 DEBUG:boto:Data: 2023-09-29T22:22:43.655 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.655 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.655 DEBUG:boto:Params: {} 2023-09-29T22:22:43.655 DEBUG:boto:Token: None 2023-09-29T22:22:43.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004172be9ca2b67877-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.698 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.699 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.699 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.699 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.699 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.699 DEBUG:boto:Method: GET 2023-09-29T22:22:43.699 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.700 DEBUG:boto:Data: 2023-09-29T22:22:43.700 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.700 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.700 DEBUG:boto:Params: {} 2023-09-29T22:22:43.700 DEBUG:boto:Token: None 2023-09-29T22:22:43.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a473527f8c8fcd3a-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.703 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.703 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.703 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.704 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.704 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.704 DEBUG:boto:Data: 2023-09-29T22:22:43.704 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.704 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.704 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.704 DEBUG:boto:Params: {} 2023-09-29T22:22:43.704 DEBUG:boto:Token: None 2023-09-29T22:22:43.705 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.705 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d83c8f457ed83b0-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.708 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.708 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.708 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.708 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.708 DEBUG:boto:Data: 2023-09-29T22:22:43.708 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.709 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.709 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.709 DEBUG:boto:Params: {} 2023-09-29T22:22:43.709 DEBUG:boto:Token: None 2023-09-29T22:22:43.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052eb5d4d68c8fc8d-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.712 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.713 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.713 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.713 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.713 DEBUG:boto:Method: GET 2023-09-29T22:22:43.713 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.713 DEBUG:boto:Data: 2023-09-29T22:22:43.713 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.713 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.713 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.713 DEBUG:boto:Params: {} 2023-09-29T22:22:43.713 DEBUG:boto:Token: None 2023-09-29T22:22:43.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.714 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ae8b490bbab76b3-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.758 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.759 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.759 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.759 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.759 DEBUG:boto:Method: GET 2023-09-29T22:22:43.759 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.759 DEBUG:boto:Data: 2023-09-29T22:22:43.759 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.759 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.760 DEBUG:boto:Params: {} 2023-09-29T22:22:43.760 DEBUG:boto:Token: None 2023-09-29T22:22:43.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f4ab22c44c4a81c-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.762 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.762 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.763 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.763 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.763 DEBUG:boto:Method: GET 2023-09-29T22:22:43.763 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.763 DEBUG:boto:Data: 2023-09-29T22:22:43.763 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.763 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.763 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.763 DEBUG:boto:Params: {} 2023-09-29T22:22:43.763 DEBUG:boto:Token: None 2023-09-29T22:22:43.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.764 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354f76bb92bafead-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.806 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.807 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.807 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.807 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.807 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.807 DEBUG:boto:Method: GET 2023-09-29T22:22:43.807 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.807 DEBUG:boto:Data: 2023-09-29T22:22:43.807 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.807 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.807 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.808 DEBUG:boto:Params: {} 2023-09-29T22:22:43.808 DEBUG:boto:Token: None 2023-09-29T22:22:43.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.808 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edace09caaecc8aa-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.810 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.811 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.811 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.811 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.811 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.811 DEBUG:boto:Method: GET 2023-09-29T22:22:43.811 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.811 DEBUG:boto:Data: 2023-09-29T22:22:43.811 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.812 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.812 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.812 DEBUG:boto:Params: {} 2023-09-29T22:22:43.812 DEBUG:boto:Token: None 2023-09-29T22:22:43.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM= 2023-09-29T22:22:43.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM=', 'Content-Length': '0'} 2023-09-29T22:22:43.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016e8ee19512c3905-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.814 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.814 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.814 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.814 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.815 DEBUG:boto:Method: GET 2023-09-29T22:22:43.815 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.815 DEBUG:boto:Data: 2023-09-29T22:22:43.815 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.815 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.815 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.815 DEBUG:boto:Params: {} 2023-09-29T22:22:43.815 DEBUG:boto:Token: None 2023-09-29T22:22:43.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.815 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM= 2023-09-29T22:22:43.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM=', 'Content-Length': '0'} 2023-09-29T22:22:43.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3d0eb5d82d83adf-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.818 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.818 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.818 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.818 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.818 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.818 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.818 DEBUG:boto:Data: 2023-09-29T22:22:43.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.819 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.819 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.819 DEBUG:boto:Params: {} 2023-09-29T22:22:43.819 DEBUG:boto:Token: None 2023-09-29T22:22:43.819 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.819 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c4ZayXeoe/kPUaJwy08/ZGxeEok= 2023-09-29T22:22:43.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c4ZayXeoe/kPUaJwy08/ZGxeEok=', 'Content-Length': '0'} 2023-09-29T22:22:43.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066ceffd9a4c3f311-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.821 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.821 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.821 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.822 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.822 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.822 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.822 DEBUG:boto:Data: 2023-09-29T22:22:43.822 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.822 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.822 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.822 DEBUG:boto:Params: {} 2023-09-29T22:22:43.822 DEBUG:boto:Token: None 2023-09-29T22:22:43.823 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.823 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c4ZayXeoe/kPUaJwy08/ZGxeEok= 2023-09-29T22:22:43.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c4ZayXeoe/kPUaJwy08/ZGxeEok=', 'Content-Length': '0'} 2023-09-29T22:22:43.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b04668bd7bf60147-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.825 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.825 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.825 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.825 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.825 DEBUG:boto:Method: GET 2023-09-29T22:22:43.825 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.825 DEBUG:boto:Data: 2023-09-29T22:22:43.825 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.826 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.826 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.826 DEBUG:boto:Params: {} 2023-09-29T22:22:43.826 DEBUG:boto:Token: None 2023-09-29T22:22:43.826 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.826 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM= 2023-09-29T22:22:43.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM=', 'Content-Length': '0'} 2023-09-29T22:22:43.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000082286bc84edb1fb-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.828 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.828 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.828 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.829 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.829 DEBUG:boto:Method: GET 2023-09-29T22:22:43.829 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.829 DEBUG:boto:Data: 2023-09-29T22:22:43.829 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.829 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.829 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.829 DEBUG:boto:Params: {} 2023-09-29T22:22:43.829 DEBUG:boto:Token: None 2023-09-29T22:22:43.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:43.830 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM= 2023-09-29T22:22:43.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HcIFXhxvpuDETzPi8H4F5yhXfGM=', 'Content-Length': '0'} 2023-09-29T22:22:43.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040e8370497487df9-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.832 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.832 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.832 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.832 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.832 DEBUG:boto:Method: GET 2023-09-29T22:22:43.832 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.833 DEBUG:boto:Data: 2023-09-29T22:22:43.833 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.833 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.833 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.833 DEBUG:boto:Params: {} 2023-09-29T22:22:43.833 DEBUG:boto:Token: None 2023-09-29T22:22:43.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.833 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e82250be2940512a-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.836 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.836 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.837 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.837 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.837 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.837 DEBUG:boto:Method: GET 2023-09-29T22:22:43.837 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.837 DEBUG:boto:Data: 2023-09-29T22:22:43.837 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.837 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.837 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.838 DEBUG:boto:Params: {} 2023-09-29T22:22:43.838 DEBUG:boto:Token: None 2023-09-29T22:22:43.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.838 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d0dc630727b822-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.841 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.841 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.841 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.841 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.841 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.842 DEBUG:boto:Data: 2023-09-29T22:22:43.842 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.842 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.842 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.842 DEBUG:boto:Params: {} 2023-09-29T22:22:43.842 DEBUG:boto:Token: None 2023-09-29T22:22:43.842 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.843 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cc206cad9919fe4-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.846 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.846 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.846 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.846 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:43.846 DEBUG:boto:Data: 2023-09-29T22:22:43.846 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.846 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.847 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.847 DEBUG:boto:Params: {} 2023-09-29T22:22:43.847 DEBUG:boto:Token: None 2023-09-29T22:22:43.847 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj 2023-09-29T22:22:43.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc= 2023-09-29T22:22:43.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YZX3M3w3lM60S/TWtgB7WaEBnuc=', 'Content-Length': '0'} 2023-09-29T22:22:43.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000984805d15fce4acd-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.851 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.851 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.851 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.851 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.851 DEBUG:boto:Method: GET 2023-09-29T22:22:43.851 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.851 DEBUG:boto:Data: 2023-09-29T22:22:43.851 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.851 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.851 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.852 DEBUG:boto:Params: {} 2023-09-29T22:22:43.852 DEBUG:boto:Token: None 2023-09-29T22:22:43.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014f791763f0006e9-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.898 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.898 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.899 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.899 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.899 DEBUG:boto:Method: GET 2023-09-29T22:22:43.899 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.899 DEBUG:boto:Data: 2023-09-29T22:22:43.899 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.899 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.899 DEBUG:boto:Params: {} 2023-09-29T22:22:43.899 DEBUG:boto:Token: None 2023-09-29T22:22:43.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:43.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE= 2023-09-29T22:22:43.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LyBbkb6j9eEGjhwDjS4R2oWV4iE=', 'Content-Length': '0'} 2023-09-29T22:22:43.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e37655fece39d284-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.902 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.902 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.902 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.902 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.902 DEBUG:boto:Method: GET 2023-09-29T22:22:43.902 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.902 DEBUG:boto:Data: 2023-09-29T22:22:43.903 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.903 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.903 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.903 DEBUG:boto:Params: {} 2023-09-29T22:22:43.903 DEBUG:boto:Token: None 2023-09-29T22:22:43.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.903 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0a84a300e9eb3db-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.946 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.947 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:43.947 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:43.947 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.947 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.947 DEBUG:boto:Method: GET 2023-09-29T22:22:43.947 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:43.948 DEBUG:boto:Data: 2023-09-29T22:22:43.948 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.948 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.948 DEBUG:boto:Params: {} 2023-09-29T22:22:43.948 DEBUG:boto:Token: None 2023-09-29T22:22:43.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/ 2023-09-29T22:22:43.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as= 2023-09-29T22:22:43.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7OJxFTX91vALT5PVKhwgPRNQ/as=', 'Content-Length': '0'} 2023-09-29T22:22:43.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33a47d1a19bbc16-0065174e33-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.951 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:43.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.951 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.952 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.952 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.952 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.952 DEBUG:boto:Method: GET 2023-09-29T22:22:43.952 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.952 DEBUG:boto:Data: 2023-09-29T22:22:43.952 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.952 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.952 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.952 DEBUG:boto:Params: {} 2023-09-29T22:22:43.953 DEBUG:boto:Token: None 2023-09-29T22:22:43.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.953 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM= 2023-09-29T22:22:43.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM=', 'Content-Length': '0'} 2023-09-29T22:22:43.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2f40879fae49c3-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.955 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.955 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.955 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.955 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.956 DEBUG:boto:Method: GET 2023-09-29T22:22:43.956 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.956 DEBUG:boto:Data: 2023-09-29T22:22:43.956 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.956 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.956 DEBUG:boto:Params: {} 2023-09-29T22:22:43.956 DEBUG:boto:Token: None 2023-09-29T22:22:43.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM= 2023-09-29T22:22:43.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM=', 'Content-Length': '0'} 2023-09-29T22:22:43.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039798b0f4f4bdf3b-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.959 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.959 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.959 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.959 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.959 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.959 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.959 DEBUG:boto:Data: 2023-09-29T22:22:43.959 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.959 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.960 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.960 DEBUG:boto:Params: {} 2023-09-29T22:22:43.960 DEBUG:boto:Token: None 2023-09-29T22:22:43.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k4qbyZ/X1g6GwLzfKeZLiD0BrFc= 2023-09-29T22:22:43.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k4qbyZ/X1g6GwLzfKeZLiD0BrFc=', 'Content-Length': '0'} 2023-09-29T22:22:43.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4499833462c2619-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.962 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.962 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.962 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.962 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.962 DEBUG:boto:Method: HEAD 2023-09-29T22:22:43.963 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.963 DEBUG:boto:Data: 2023-09-29T22:22:43.963 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.963 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.963 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.963 DEBUG:boto:Params: {} 2023-09-29T22:22:43.963 DEBUG:boto:Token: None 2023-09-29T22:22:43.963 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.963 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k4qbyZ/X1g6GwLzfKeZLiD0BrFc= 2023-09-29T22:22:43.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k4qbyZ/X1g6GwLzfKeZLiD0BrFc=', 'Content-Length': '0'} 2023-09-29T22:22:43.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ac467455da02059-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:43.966 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.966 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.966 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.966 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.966 DEBUG:boto:Method: GET 2023-09-29T22:22:43.966 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.966 DEBUG:boto:Data: 2023-09-29T22:22:43.966 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.967 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:43.967 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.967 DEBUG:boto:Params: {} 2023-09-29T22:22:43.967 DEBUG:boto:Token: None 2023-09-29T22:22:43.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM= 2023-09-29T22:22:43.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM=', 'Content-Length': '0'} 2023-09-29T22:22:43.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085fe64b1691244cb-0065174e33-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:43.969 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:43.970 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:43.970 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.970 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.970 DEBUG:boto:Method: GET 2023-09-29T22:22:43.970 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.970 DEBUG:boto:Data: 2023-09-29T22:22:43.970 DEBUG:boto:Headers: {} 2023-09-29T22:22:43.970 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:43.970 DEBUG:boto:Port: 8000 2023-09-29T22:22:43.971 DEBUG:boto:Params: {} 2023-09-29T22:22:43.971 DEBUG:boto:Token: None 2023-09-29T22:22:43.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:43 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:43.971 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM= 2023-09-29T22:22:43.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2GhvKQS3LvTJ3iOkpwJPj7EXMfM=', 'Content-Length': '0'} 2023-09-29T22:22:43.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eea2b6093e330083-0065174e33-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.014 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.014 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.015 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.015 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.015 DEBUG:boto:Method: GET 2023-09-29T22:22:44.015 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.015 DEBUG:boto:Data: 2023-09-29T22:22:44.015 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.015 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.015 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.015 DEBUG:boto:Params: {} 2023-09-29T22:22:44.015 DEBUG:boto:Token: None 2023-09-29T22:22:44.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc3eb2bf3f9b1c42-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.018 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.019 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.019 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.019 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.019 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.019 DEBUG:boto:Method: GET 2023-09-29T22:22:44.019 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.020 DEBUG:boto:Data: 2023-09-29T22:22:44.020 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.020 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.020 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.020 DEBUG:boto:Params: {} 2023-09-29T22:22:44.020 DEBUG:boto:Token: None 2023-09-29T22:22:44.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704614aa50d6f431-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.023 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.023 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.023 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.024 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.024 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.024 DEBUG:boto:Data: 2023-09-29T22:22:44.024 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.024 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.024 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.024 DEBUG:boto:Params: {} 2023-09-29T22:22:44.024 DEBUG:boto:Token: None 2023-09-29T22:22:44.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.025 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050c3e52696e44a11-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.028 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.028 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.028 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.028 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.028 DEBUG:boto:Data: 2023-09-29T22:22:44.028 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.029 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.029 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.029 DEBUG:boto:Params: {} 2023-09-29T22:22:44.029 DEBUG:boto:Token: None 2023-09-29T22:22:44.029 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053004933a38bdf1e-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.032 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.033 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.033 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.033 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.033 DEBUG:boto:Method: GET 2023-09-29T22:22:44.033 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.033 DEBUG:boto:Data: 2023-09-29T22:22:44.033 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.033 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.033 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.033 DEBUG:boto:Params: {} 2023-09-29T22:22:44.034 DEBUG:boto:Token: None 2023-09-29T22:22:44.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053234cdb1889a42d-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.078 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.082 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.082 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.082 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.082 DEBUG:boto:Method: GET 2023-09-29T22:22:44.082 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.082 DEBUG:boto:Data: 2023-09-29T22:22:44.082 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.082 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.083 DEBUG:boto:Params: {} 2023-09-29T22:22:44.083 DEBUG:boto:Token: None 2023-09-29T22:22:44.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.083 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000809aebd8f37a4589-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.086 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.086 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.086 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.086 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.086 DEBUG:boto:Method: GET 2023-09-29T22:22:44.086 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.086 DEBUG:boto:Data: 2023-09-29T22:22:44.086 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.087 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.087 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.087 DEBUG:boto:Params: {} 2023-09-29T22:22:44.087 DEBUG:boto:Token: None 2023-09-29T22:22:44.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129eadc514ca5eb1-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.090 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.090 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.091 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.091 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.091 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.091 DEBUG:boto:Method: GET 2023-09-29T22:22:44.091 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.091 DEBUG:boto:Data: 2023-09-29T22:22:44.091 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.091 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.091 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.091 DEBUG:boto:Params: {} 2023-09-29T22:22:44.092 DEBUG:boto:Token: None 2023-09-29T22:22:44.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.092 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3751ccb6dc5a82-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.095 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.095 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.095 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.095 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.096 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.096 DEBUG:boto:Method: GET 2023-09-29T22:22:44.096 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.096 DEBUG:boto:Data: 2023-09-29T22:22:44.096 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.096 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.096 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.096 DEBUG:boto:Params: {} 2023-09-29T22:22:44.096 DEBUG:boto:Token: None 2023-09-29T22:22:44.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.097 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns= 2023-09-29T22:22:44.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns=', 'Content-Length': '0'} 2023-09-29T22:22:44.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dda0cafad85c9160-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.099 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.099 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.099 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.099 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.099 DEBUG:boto:Method: GET 2023-09-29T22:22:44.099 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.099 DEBUG:boto:Data: 2023-09-29T22:22:44.099 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.100 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.100 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.100 DEBUG:boto:Params: {} 2023-09-29T22:22:44.100 DEBUG:boto:Token: None 2023-09-29T22:22:44.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.100 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns= 2023-09-29T22:22:44.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns=', 'Content-Length': '0'} 2023-09-29T22:22:44.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043103fd5335dbdf4-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.146 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.146 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.147 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.147 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.147 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.147 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.147 DEBUG:boto:Data: 2023-09-29T22:22:44.147 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.147 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.147 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.147 DEBUG:boto:Params: {} 2023-09-29T22:22:44.148 DEBUG:boto:Token: None 2023-09-29T22:22:44.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XSQ2vXoPdhpIhG5hjgmIoc5k5pg= 2023-09-29T22:22:44.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XSQ2vXoPdhpIhG5hjgmIoc5k5pg=', 'Content-Length': '0'} 2023-09-29T22:22:44.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad02f2e80529adb7-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.150 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.150 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.150 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.150 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.151 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.151 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.151 DEBUG:boto:Data: 2023-09-29T22:22:44.151 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.151 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.151 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.151 DEBUG:boto:Params: {} 2023-09-29T22:22:44.151 DEBUG:boto:Token: None 2023-09-29T22:22:44.151 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XSQ2vXoPdhpIhG5hjgmIoc5k5pg= 2023-09-29T22:22:44.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XSQ2vXoPdhpIhG5hjgmIoc5k5pg=', 'Content-Length': '0'} 2023-09-29T22:22:44.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c5999d96bcb220-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.154 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.154 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.154 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.154 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.154 DEBUG:boto:Method: GET 2023-09-29T22:22:44.154 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.154 DEBUG:boto:Data: 2023-09-29T22:22:44.155 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.155 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.155 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.155 DEBUG:boto:Params: {} 2023-09-29T22:22:44.155 DEBUG:boto:Token: None 2023-09-29T22:22:44.155 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns= 2023-09-29T22:22:44.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns=', 'Content-Length': '0'} 2023-09-29T22:22:44.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a9951ce2544bbe-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.198 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.198 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.199 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.199 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.199 DEBUG:boto:Method: GET 2023-09-29T22:22:44.199 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.199 DEBUG:boto:Data: 2023-09-29T22:22:44.199 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.199 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.199 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.199 DEBUG:boto:Params: {} 2023-09-29T22:22:44.199 DEBUG:boto:Token: None 2023-09-29T22:22:44.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:44.200 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns= 2023-09-29T22:22:44.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DtVMKEY3U26aI/9GDlBz8UWLpns=', 'Content-Length': '0'} 2023-09-29T22:22:44.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055a1d8a792a7b559-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.202 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.202 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.202 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.202 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.202 DEBUG:boto:Method: GET 2023-09-29T22:22:44.203 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.203 DEBUG:boto:Data: 2023-09-29T22:22:44.203 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.203 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.203 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.203 DEBUG:boto:Params: {} 2023-09-29T22:22:44.203 DEBUG:boto:Token: None 2023-09-29T22:22:44.203 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.203 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b02faa7e5208c5fa-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.206 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.206 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.207 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.207 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.207 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.207 DEBUG:boto:Method: GET 2023-09-29T22:22:44.207 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.207 DEBUG:boto:Data: 2023-09-29T22:22:44.207 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.207 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.207 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.208 DEBUG:boto:Params: {} 2023-09-29T22:22:44.208 DEBUG:boto:Token: None 2023-09-29T22:22:44.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.208 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd20757047908ed-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.211 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.211 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.211 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.211 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.211 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.211 DEBUG:boto:Data: 2023-09-29T22:22:44.211 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.212 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.212 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.212 DEBUG:boto:Params: {} 2023-09-29T22:22:44.212 DEBUG:boto:Token: None 2023-09-29T22:22:44.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a09c821483d0c95c-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.216 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.216 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.216 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.216 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.216 DEBUG:boto:Data: 2023-09-29T22:22:44.216 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.216 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.216 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.216 DEBUG:boto:Params: {} 2023-09-29T22:22:44.217 DEBUG:boto:Token: None 2023-09-29T22:22:44.217 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.217 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a1f130ed3bc39b8-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.220 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.220 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.220 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.220 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.220 DEBUG:boto:Method: GET 2023-09-29T22:22:44.221 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.221 DEBUG:boto:Data: 2023-09-29T22:22:44.221 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.221 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.221 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.221 DEBUG:boto:Params: {} 2023-09-29T22:22:44.221 DEBUG:boto:Token: None 2023-09-29T22:22:44.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.221 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000510e157d231286de-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.266 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.266 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.267 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.267 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.267 DEBUG:boto:Method: GET 2023-09-29T22:22:44.267 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.267 DEBUG:boto:Data: 2023-09-29T22:22:44.267 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.267 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.267 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.267 DEBUG:boto:Params: {} 2023-09-29T22:22:44.267 DEBUG:boto:Token: None 2023-09-29T22:22:44.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f1f1b7eea125e78-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.270 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.270 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.270 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.270 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.270 DEBUG:boto:Method: GET 2023-09-29T22:22:44.270 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.270 DEBUG:boto:Data: 2023-09-29T22:22:44.271 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.271 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.271 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.271 DEBUG:boto:Params: {} 2023-09-29T22:22:44.271 DEBUG:boto:Token: None 2023-09-29T22:22:44.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003133f5424f86a27d-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.274 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.274 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.274 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.274 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.274 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.274 DEBUG:boto:Method: GET 2023-09-29T22:22:44.274 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.275 DEBUG:boto:Data: 2023-09-29T22:22:44.275 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.275 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.275 DEBUG:boto:Params: {} 2023-09-29T22:22:44.275 DEBUG:boto:Token: None 2023-09-29T22:22:44.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027fbaeafd924f989-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.278 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.278 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.278 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.279 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.279 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.279 DEBUG:boto:Method: GET 2023-09-29T22:22:44.279 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.279 DEBUG:boto:Data: 2023-09-29T22:22:44.279 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.279 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.279 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.279 DEBUG:boto:Params: {} 2023-09-29T22:22:44.279 DEBUG:boto:Token: None 2023-09-29T22:22:44.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.280 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg= 2023-09-29T22:22:44.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg=', 'Content-Length': '0'} 2023-09-29T22:22:44.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fd3ace024a74d1a-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.322 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.322 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.323 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.323 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.323 DEBUG:boto:Method: GET 2023-09-29T22:22:44.323 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.323 DEBUG:boto:Data: 2023-09-29T22:22:44.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.323 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.323 DEBUG:boto:Params: {} 2023-09-29T22:22:44.323 DEBUG:boto:Token: None 2023-09-29T22:22:44.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg= 2023-09-29T22:22:44.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg=', 'Content-Length': '0'} 2023-09-29T22:22:44.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004392445a61431beb-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.326 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.326 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.326 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.326 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.326 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.326 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.326 DEBUG:boto:Data: 2023-09-29T22:22:44.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.327 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.327 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.327 DEBUG:boto:Params: {} 2023-09-29T22:22:44.327 DEBUG:boto:Token: None 2023-09-29T22:22:44.327 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EvIURO412pTiPZlXi2NV4pG6IGI= 2023-09-29T22:22:44.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EvIURO412pTiPZlXi2NV4pG6IGI=', 'Content-Length': '0'} 2023-09-29T22:22:44.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b48c5ab082fd4ee7-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.329 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.329 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.329 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.329 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.330 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.330 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.330 DEBUG:boto:Data: 2023-09-29T22:22:44.330 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.330 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.330 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.330 DEBUG:boto:Params: {} 2023-09-29T22:22:44.330 DEBUG:boto:Token: None 2023-09-29T22:22:44.330 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EvIURO412pTiPZlXi2NV4pG6IGI= 2023-09-29T22:22:44.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EvIURO412pTiPZlXi2NV4pG6IGI=', 'Content-Length': '0'} 2023-09-29T22:22:44.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092aea9ed47740927-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.332 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.333 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.333 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.333 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.333 DEBUG:boto:Method: GET 2023-09-29T22:22:44.333 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.333 DEBUG:boto:Data: 2023-09-29T22:22:44.333 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.333 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.333 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.333 DEBUG:boto:Params: {} 2023-09-29T22:22:44.333 DEBUG:boto:Token: None 2023-09-29T22:22:44.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.334 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg= 2023-09-29T22:22:44.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg=', 'Content-Length': '0'} 2023-09-29T22:22:44.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c6ee34f165fd2b5-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.378 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.378 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.379 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.379 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.379 DEBUG:boto:Method: GET 2023-09-29T22:22:44.379 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.379 DEBUG:boto:Data: 2023-09-29T22:22:44.379 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.379 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.379 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.379 DEBUG:boto:Params: {} 2023-09-29T22:22:44.380 DEBUG:boto:Token: None 2023-09-29T22:22:44.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:44.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg= 2023-09-29T22:22:44.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnYA2oDsDv9IDqQTdy76bIrhewg=', 'Content-Length': '0'} 2023-09-29T22:22:44.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6730c651c9d131f-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.382 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.382 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.382 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.382 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.382 DEBUG:boto:Method: GET 2023-09-29T22:22:44.382 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.383 DEBUG:boto:Data: 2023-09-29T22:22:44.383 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.383 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.383 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.383 DEBUG:boto:Params: {} 2023-09-29T22:22:44.383 DEBUG:boto:Token: None 2023-09-29T22:22:44.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f0fde80b973ccce-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.426 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.427 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.427 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.427 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.427 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.427 DEBUG:boto:Method: GET 2023-09-29T22:22:44.427 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.427 DEBUG:boto:Data: 2023-09-29T22:22:44.428 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.428 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.428 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.428 DEBUG:boto:Params: {} 2023-09-29T22:22:44.428 DEBUG:boto:Token: None 2023-09-29T22:22:44.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e5782792d115b66-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.431 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.431 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.432 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.432 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.432 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.432 DEBUG:boto:Data: 2023-09-29T22:22:44.432 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.432 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.432 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.432 DEBUG:boto:Params: {} 2023-09-29T22:22:44.432 DEBUG:boto:Token: None 2023-09-29T22:22:44.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.433 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6da77d1c67a3384-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.436 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.436 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.436 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.436 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.436 DEBUG:boto:Data: 2023-09-29T22:22:44.436 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.437 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.437 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.437 DEBUG:boto:Params: {} 2023-09-29T22:22:44.437 DEBUG:boto:Token: None 2023-09-29T22:22:44.437 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.437 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed644999f0aada27-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.441 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.441 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.441 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.441 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.441 DEBUG:boto:Method: GET 2023-09-29T22:22:44.442 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.442 DEBUG:boto:Data: 2023-09-29T22:22:44.442 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.442 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.442 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.442 DEBUG:boto:Params: {} 2023-09-29T22:22:44.442 DEBUG:boto:Token: None 2023-09-29T22:22:44.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.442 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a06f3064c5ea1c4d-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.489 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.490 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.490 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.490 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.490 DEBUG:boto:Method: GET 2023-09-29T22:22:44.490 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.490 DEBUG:boto:Data: 2023-09-29T22:22:44.490 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.490 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.491 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.491 DEBUG:boto:Params: {} 2023-09-29T22:22:44.491 DEBUG:boto:Token: None 2023-09-29T22:22:44.491 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f037d312014a166-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.493 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.493 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.494 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.494 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.494 DEBUG:boto:Method: GET 2023-09-29T22:22:44.494 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.494 DEBUG:boto:Data: 2023-09-29T22:22:44.494 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.494 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.494 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.494 DEBUG:boto:Params: {} 2023-09-29T22:22:44.495 DEBUG:boto:Token: None 2023-09-29T22:22:44.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd67e6f16b3a2c5f-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.538 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.539 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.539 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.539 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.539 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.539 DEBUG:boto:Method: GET 2023-09-29T22:22:44.539 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.539 DEBUG:boto:Data: 2023-09-29T22:22:44.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.540 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.540 DEBUG:boto:Params: {} 2023-09-29T22:22:44.540 DEBUG:boto:Token: None 2023-09-29T22:22:44.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075aa4ce5a8956bc6-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.543 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.543 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.544 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.544 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.544 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.544 DEBUG:boto:Method: GET 2023-09-29T22:22:44.544 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.544 DEBUG:boto:Data: 2023-09-29T22:22:44.544 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.544 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.544 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.544 DEBUG:boto:Params: {} 2023-09-29T22:22:44.544 DEBUG:boto:Token: None 2023-09-29T22:22:44.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.545 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg= 2023-09-29T22:22:44.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg=', 'Content-Length': '0'} 2023-09-29T22:22:44.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f95527012596dcd2-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.547 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.547 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.547 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.547 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.547 DEBUG:boto:Method: GET 2023-09-29T22:22:44.547 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.547 DEBUG:boto:Data: 2023-09-29T22:22:44.548 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.548 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.548 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.548 DEBUG:boto:Params: {} 2023-09-29T22:22:44.548 DEBUG:boto:Token: None 2023-09-29T22:22:44.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.548 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg= 2023-09-29T22:22:44.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg=', 'Content-Length': '0'} 2023-09-29T22:22:44.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b6b8279d0d93793-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.550 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.550 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.551 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.551 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.551 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.551 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.551 DEBUG:boto:Data: 2023-09-29T22:22:44.551 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.551 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.551 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.551 DEBUG:boto:Params: {} 2023-09-29T22:22:44.552 DEBUG:boto:Token: None 2023-09-29T22:22:44.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91wWLmOiOObqnH9220jM0WJLcHU= 2023-09-29T22:22:44.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91wWLmOiOObqnH9220jM0WJLcHU=', 'Content-Length': '0'} 2023-09-29T22:22:44.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd0db3b5c8ef35c8-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.554 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.554 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.554 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.554 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.554 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.554 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.554 DEBUG:boto:Data: 2023-09-29T22:22:44.554 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.554 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.555 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.555 DEBUG:boto:Params: {} 2023-09-29T22:22:44.555 DEBUG:boto:Token: None 2023-09-29T22:22:44.555 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.555 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:91wWLmOiOObqnH9220jM0WJLcHU= 2023-09-29T22:22:44.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:91wWLmOiOObqnH9220jM0WJLcHU=', 'Content-Length': '0'} 2023-09-29T22:22:44.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097c30eb2aeea3ff4-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.557 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.557 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.557 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.558 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.558 DEBUG:boto:Method: GET 2023-09-29T22:22:44.558 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.558 DEBUG:boto:Data: 2023-09-29T22:22:44.558 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.558 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.558 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.558 DEBUG:boto:Params: {} 2023-09-29T22:22:44.558 DEBUG:boto:Token: None 2023-09-29T22:22:44.559 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.559 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg= 2023-09-29T22:22:44.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg=', 'Content-Length': '0'} 2023-09-29T22:22:44.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3b07b97a045fbfa-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.561 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.561 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.561 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.561 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.561 DEBUG:boto:Method: GET 2023-09-29T22:22:44.561 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.562 DEBUG:boto:Data: 2023-09-29T22:22:44.562 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.562 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.562 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.562 DEBUG:boto:Params: {} 2023-09-29T22:22:44.562 DEBUG:boto:Token: None 2023-09-29T22:22:44.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:44.562 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg= 2023-09-29T22:22:44.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:onH6ScPMwXe4Z4PaQBXrZaY1Dxg=', 'Content-Length': '0'} 2023-09-29T22:22:44.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000be2751051612d3-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.564 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.565 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.565 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.565 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.565 DEBUG:boto:Method: GET 2023-09-29T22:22:44.565 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.565 DEBUG:boto:Data: 2023-09-29T22:22:44.565 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.565 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.565 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.565 DEBUG:boto:Params: {} 2023-09-29T22:22:44.566 DEBUG:boto:Token: None 2023-09-29T22:22:44.566 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.566 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70acdd150753089-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.569 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.569 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.569 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.569 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.569 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.569 DEBUG:boto:Method: GET 2023-09-29T22:22:44.570 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.570 DEBUG:boto:Data: 2023-09-29T22:22:44.570 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.570 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.570 DEBUG:boto:Params: {} 2023-09-29T22:22:44.570 DEBUG:boto:Token: None 2023-09-29T22:22:44.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.570 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a90a1c8f0995e78-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.573 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.573 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.573 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.574 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.574 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.574 DEBUG:boto:Data: 2023-09-29T22:22:44.574 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.574 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.574 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.574 DEBUG:boto:Params: {} 2023-09-29T22:22:44.574 DEBUG:boto:Token: None 2023-09-29T22:22:44.574 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.575 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000185010367f70edd7-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.578 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.578 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.578 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.578 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.579 DEBUG:boto:Data: 2023-09-29T22:22:44.579 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.579 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.579 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.579 DEBUG:boto:Params: {} 2023-09-29T22:22:44.579 DEBUG:boto:Token: None 2023-09-29T22:22:44.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076280597a26fe5bb-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.583 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.583 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.583 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.583 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.583 DEBUG:boto:Method: GET 2023-09-29T22:22:44.583 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.583 DEBUG:boto:Data: 2023-09-29T22:22:44.584 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.584 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.584 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.584 DEBUG:boto:Params: {} 2023-09-29T22:22:44.584 DEBUG:boto:Token: None 2023-09-29T22:22:44.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ffcf2934f404da8-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.626 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.627 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.627 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.627 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.627 DEBUG:boto:Method: GET 2023-09-29T22:22:44.627 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.627 DEBUG:boto:Data: 2023-09-29T22:22:44.627 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.627 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.627 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.627 DEBUG:boto:Params: {} 2023-09-29T22:22:44.627 DEBUG:boto:Token: None 2023-09-29T22:22:44.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.628 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7dc314132de3fc0-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.630 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.630 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.630 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.630 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.630 DEBUG:boto:Method: GET 2023-09-29T22:22:44.631 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.631 DEBUG:boto:Data: 2023-09-29T22:22:44.631 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.631 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.631 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.631 DEBUG:boto:Params: {} 2023-09-29T22:22:44.631 DEBUG:boto:Token: None 2023-09-29T22:22:44.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.631 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff844dae1002982c-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.674 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.675 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.675 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.675 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.675 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.675 DEBUG:boto:Method: GET 2023-09-29T22:22:44.675 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.675 DEBUG:boto:Data: 2023-09-29T22:22:44.675 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.675 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.676 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.676 DEBUG:boto:Params: {} 2023-09-29T22:22:44.676 DEBUG:boto:Token: None 2023-09-29T22:22:44.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551fd165635ad5c0-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.679 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.679 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.679 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.679 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.680 DEBUG:boto:Method: GET 2023-09-29T22:22:44.680 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.680 DEBUG:boto:Data: 2023-09-29T22:22:44.680 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.680 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.680 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.680 DEBUG:boto:Params: {} 2023-09-29T22:22:44.680 DEBUG:boto:Token: None 2023-09-29T22:22:44.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc= 2023-09-29T22:22:44.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc=', 'Content-Length': '0'} 2023-09-29T22:22:44.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082b11b1c68137bb0-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.683 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.683 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.683 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.683 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.683 DEBUG:boto:Method: GET 2023-09-29T22:22:44.683 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.683 DEBUG:boto:Data: 2023-09-29T22:22:44.683 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.684 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.684 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.684 DEBUG:boto:Params: {} 2023-09-29T22:22:44.684 DEBUG:boto:Token: None 2023-09-29T22:22:44.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc= 2023-09-29T22:22:44.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc=', 'Content-Length': '0'} 2023-09-29T22:22:44.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002ddb5ce481f3c22-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.686 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.686 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.686 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.686 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.687 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.687 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.687 DEBUG:boto:Data: 2023-09-29T22:22:44.687 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.687 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.687 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.687 DEBUG:boto:Params: {} 2023-09-29T22:22:44.687 DEBUG:boto:Token: None 2023-09-29T22:22:44.687 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.688 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWjUFBgVTiPPFSVwR40ribk78Zs= 2023-09-29T22:22:44.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWjUFBgVTiPPFSVwR40ribk78Zs=', 'Content-Length': '0'} 2023-09-29T22:22:44.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ef38d5e33bfb138-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.690 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.690 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.690 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.690 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.690 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.690 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.694 DEBUG:boto:Data: 2023-09-29T22:22:44.694 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.694 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.694 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.694 DEBUG:boto:Params: {} 2023-09-29T22:22:44.694 DEBUG:boto:Token: None 2023-09-29T22:22:44.695 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.695 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OWjUFBgVTiPPFSVwR40ribk78Zs= 2023-09-29T22:22:44.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OWjUFBgVTiPPFSVwR40ribk78Zs=', 'Content-Length': '0'} 2023-09-29T22:22:44.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000262fe7738d49ac5a-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.697 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.697 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.697 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.697 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.697 DEBUG:boto:Method: GET 2023-09-29T22:22:44.697 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.698 DEBUG:boto:Data: 2023-09-29T22:22:44.698 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.698 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.698 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.698 DEBUG:boto:Params: {} 2023-09-29T22:22:44.698 DEBUG:boto:Token: None 2023-09-29T22:22:44.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.698 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc= 2023-09-29T22:22:44.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc=', 'Content-Length': '0'} 2023-09-29T22:22:44.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cc6b76e415059ac-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.742 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.742 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.743 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.743 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.743 DEBUG:boto:Method: GET 2023-09-29T22:22:44.743 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.743 DEBUG:boto:Data: 2023-09-29T22:22:44.743 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.743 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.743 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.743 DEBUG:boto:Params: {} 2023-09-29T22:22:44.743 DEBUG:boto:Token: None 2023-09-29T22:22:44.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:44.744 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc= 2023-09-29T22:22:44.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Up8Kmo+arM/+ixwO26lZ5phQ0qc=', 'Content-Length': '0'} 2023-09-29T22:22:44.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb49b0675895c84f-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.746 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.746 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.746 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.746 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.746 DEBUG:boto:Method: GET 2023-09-29T22:22:44.747 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.747 DEBUG:boto:Data: 2023-09-29T22:22:44.747 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.747 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.747 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.747 DEBUG:boto:Params: {} 2023-09-29T22:22:44.747 DEBUG:boto:Token: None 2023-09-29T22:22:44.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.747 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2296b3591d8117d-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.794 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.795 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.795 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.795 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.795 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.795 DEBUG:boto:Method: GET 2023-09-29T22:22:44.795 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.795 DEBUG:boto:Data: 2023-09-29T22:22:44.795 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.796 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.796 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.796 DEBUG:boto:Params: {} 2023-09-29T22:22:44.796 DEBUG:boto:Token: None 2023-09-29T22:22:44.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.796 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99092305e9c832c-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.799 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.799 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.799 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.799 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.799 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.800 DEBUG:boto:Data: 2023-09-29T22:22:44.800 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.800 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.800 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.800 DEBUG:boto:Params: {} 2023-09-29T22:22:44.800 DEBUG:boto:Token: None 2023-09-29T22:22:44.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007134205f65a3f677-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.804 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.804 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.804 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.804 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:44.804 DEBUG:boto:Data: 2023-09-29T22:22:44.804 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.804 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.804 DEBUG:boto:Params: {} 2023-09-29T22:22:44.805 DEBUG:boto:Token: None 2023-09-29T22:22:44.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj 2023-09-29T22:22:44.805 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ= 2023-09-29T22:22:44.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Fvt/HG05UvXQ8k5ZUTrPySDPDrQ=', 'Content-Length': '0'} 2023-09-29T22:22:44.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eedcd6320d8b13b-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.808 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.808 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.808 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.809 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.809 DEBUG:boto:Method: GET 2023-09-29T22:22:44.809 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.809 DEBUG:boto:Data: 2023-09-29T22:22:44.809 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.809 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.809 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.809 DEBUG:boto:Params: {} 2023-09-29T22:22:44.809 DEBUG:boto:Token: None 2023-09-29T22:22:44.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.810 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000506701c073ad5c36-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.854 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.855 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.855 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.855 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.855 DEBUG:boto:Method: GET 2023-09-29T22:22:44.855 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.855 DEBUG:boto:Data: 2023-09-29T22:22:44.855 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.855 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.855 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.855 DEBUG:boto:Params: {} 2023-09-29T22:22:44.856 DEBUG:boto:Token: None 2023-09-29T22:22:44.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:44.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc= 2023-09-29T22:22:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HhNrSfYTRVYagdpC+I0K6hjUPmc=', 'Content-Length': '0'} 2023-09-29T22:22:44.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000565cb02ae4f24c9c-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.858 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.858 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.858 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.858 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.859 DEBUG:boto:Method: GET 2023-09-29T22:22:44.859 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.859 DEBUG:boto:Data: 2023-09-29T22:22:44.859 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.859 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.859 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.859 DEBUG:boto:Params: {} 2023-09-29T22:22:44.859 DEBUG:boto:Token: None 2023-09-29T22:22:44.859 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.860 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56bc04b148ebdf5-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.902 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.903 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:44.903 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:44.903 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.903 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.903 DEBUG:boto:Method: GET 2023-09-29T22:22:44.903 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:44.903 DEBUG:boto:Data: 2023-09-29T22:22:44.903 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.904 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.904 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.904 DEBUG:boto:Params: {} 2023-09-29T22:22:44.904 DEBUG:boto:Token: None 2023-09-29T22:22:44.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/ 2023-09-29T22:22:44.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc= 2023-09-29T22:22:44.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ur6XI9nSeCQWTNAK25FF448fsfc=', 'Content-Length': '0'} 2023-09-29T22:22:44.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249dc63f520e3410-0065174e34-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.907 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:44.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.907 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.907 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.907 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.908 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.908 DEBUG:boto:Method: GET 2023-09-29T22:22:44.908 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.908 DEBUG:boto:Data: 2023-09-29T22:22:44.908 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.908 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.908 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.908 DEBUG:boto:Params: {} 2023-09-29T22:22:44.908 DEBUG:boto:Token: None 2023-09-29T22:22:44.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.909 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM= 2023-09-29T22:22:44.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM=', 'Content-Length': '0'} 2023-09-29T22:22:44.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044decda06ba19d7c-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.954 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.954 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.955 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.955 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.955 DEBUG:boto:Method: GET 2023-09-29T22:22:44.955 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.955 DEBUG:boto:Data: 2023-09-29T22:22:44.955 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.955 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.955 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.955 DEBUG:boto:Params: {} 2023-09-29T22:22:44.955 DEBUG:boto:Token: None 2023-09-29T22:22:44.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM= 2023-09-29T22:22:44.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM=', 'Content-Length': '0'} 2023-09-29T22:22:44.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dadfd3ea4e27b248-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.958 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.958 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.958 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.958 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.959 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.959 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.959 DEBUG:boto:Data: 2023-09-29T22:22:44.959 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.959 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.959 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.959 DEBUG:boto:Params: {} 2023-09-29T22:22:44.959 DEBUG:boto:Token: None 2023-09-29T22:22:44.959 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BmdlYoQbEkZgoiAsfhARWVVcbTE= 2023-09-29T22:22:44.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BmdlYoQbEkZgoiAsfhARWVVcbTE=', 'Content-Length': '0'} 2023-09-29T22:22:44.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001deb030ce87ee918-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.962 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.962 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.962 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.962 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.962 DEBUG:boto:Method: HEAD 2023-09-29T22:22:44.962 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.962 DEBUG:boto:Data: 2023-09-29T22:22:44.962 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.962 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:44.963 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.963 DEBUG:boto:Params: {} 2023-09-29T22:22:44.963 DEBUG:boto:Token: None 2023-09-29T22:22:44.963 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.963 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BmdlYoQbEkZgoiAsfhARWVVcbTE= 2023-09-29T22:22:44.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BmdlYoQbEkZgoiAsfhARWVVcbTE=', 'Content-Length': '0'} 2023-09-29T22:22:44.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7590c59f1135f74-0065174e34-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:44.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:44.965 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:44.966 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:44.966 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.966 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.966 DEBUG:boto:Method: GET 2023-09-29T22:22:44.966 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.966 DEBUG:boto:Data: 2023-09-29T22:22:44.966 DEBUG:boto:Headers: {} 2023-09-29T22:22:44.966 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:44.966 DEBUG:boto:Port: 8000 2023-09-29T22:22:44.966 DEBUG:boto:Params: {} 2023-09-29T22:22:44.967 DEBUG:boto:Token: None 2023-09-29T22:22:44.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:44 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:44.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM= 2023-09-29T22:22:44.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:44 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oBh7DbkHUcrENmlcr/fvZZGuxYM=', 'Content-Length': '0'} 2023-09-29T22:22:44.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf0bc394de7f2cef-0065174e34-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.010 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.010 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.011 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:45.011 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:45.011 DEBUG:boto:Method: GET 2023-09-29T22:22:45.011 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:45.011 DEBUG:boto:Data: 2023-09-29T22:22:45.011 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.011 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.011 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.011 DEBUG:boto:Params: {} 2023-09-29T22:22:45.011 DEBUG:boto:Token: None 2023-09-29T22:22:45.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:45.012 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8CkNlG8Ow6HuIB1FCvZ2s8BP/dI= 2023-09-29T22:22:45.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8CkNlG8Ow6HuIB1FCvZ2s8BP/dI=', 'Content-Length': '0'} 2023-09-29T22:22:45.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7f457f7b5a81f91-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.014 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.014 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.014 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.014 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.014 DEBUG:boto:Method: GET 2023-09-29T22:22:45.014 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.014 DEBUG:boto:Data: 2023-09-29T22:22:45.014 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.014 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.015 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.015 DEBUG:boto:Params: {} 2023-09-29T22:22:45.015 DEBUG:boto:Token: None 2023-09-29T22:22:45.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31d180ce9bb03cb-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.058 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.059 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.059 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.059 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.059 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.059 DEBUG:boto:Method: GET 2023-09-29T22:22:45.059 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.059 DEBUG:boto:Data: 2023-09-29T22:22:45.059 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.060 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.060 DEBUG:boto:Params: {} 2023-09-29T22:22:45.060 DEBUG:boto:Token: None 2023-09-29T22:22:45.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc7524b6ebc37a2-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.063 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.063 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.063 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.063 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.064 DEBUG:boto:Data: 2023-09-29T22:22:45.064 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.064 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.064 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.064 DEBUG:boto:Params: {} 2023-09-29T22:22:45.064 DEBUG:boto:Token: None 2023-09-29T22:22:45.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097e6130fa44519fa-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.068 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.068 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.068 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.068 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.068 DEBUG:boto:Data: 2023-09-29T22:22:45.068 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.068 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.068 DEBUG:boto:Params: {} 2023-09-29T22:22:45.068 DEBUG:boto:Token: None 2023-09-29T22:22:45.069 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0be5cc52082f623-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.072 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.072 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.072 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.072 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.072 DEBUG:boto:Method: GET 2023-09-29T22:22:45.072 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.073 DEBUG:boto:Data: 2023-09-29T22:22:45.073 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.073 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.073 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.073 DEBUG:boto:Params: {} 2023-09-29T22:22:45.073 DEBUG:boto:Token: None 2023-09-29T22:22:45.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c55a88a5e01b8b42-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.118 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.118 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.119 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.119 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.119 DEBUG:boto:Method: GET 2023-09-29T22:22:45.119 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.119 DEBUG:boto:Data: 2023-09-29T22:22:45.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.119 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.119 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.119 DEBUG:boto:Params: {} 2023-09-29T22:22:45.119 DEBUG:boto:Token: None 2023-09-29T22:22:45.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7553d87fd231a24-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.122 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.122 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.122 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.123 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.123 DEBUG:boto:Method: GET 2023-09-29T22:22:45.123 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.123 DEBUG:boto:Data: 2023-09-29T22:22:45.123 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.123 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.123 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.123 DEBUG:boto:Params: {} 2023-09-29T22:22:45.123 DEBUG:boto:Token: None 2023-09-29T22:22:45.123 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ab03ae0e51c855-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.170 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.171 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.171 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.171 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.171 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.171 DEBUG:boto:Method: GET 2023-09-29T22:22:45.171 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.171 DEBUG:boto:Data: 2023-09-29T22:22:45.171 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.171 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.172 DEBUG:boto:Params: {} 2023-09-29T22:22:45.172 DEBUG:boto:Token: None 2023-09-29T22:22:45.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc75da8108e49ee9-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.175 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.175 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.175 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.175 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.175 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.176 DEBUG:boto:Method: GET 2023-09-29T22:22:45.176 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.176 DEBUG:boto:Data: 2023-09-29T22:22:45.176 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.176 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.176 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.176 DEBUG:boto:Params: {} 2023-09-29T22:22:45.176 DEBUG:boto:Token: None 2023-09-29T22:22:45.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk= 2023-09-29T22:22:45.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk=', 'Content-Length': '0'} 2023-09-29T22:22:45.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000988fa5141b015bcf-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.179 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.179 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.179 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.179 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.179 DEBUG:boto:Method: GET 2023-09-29T22:22:45.179 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.179 DEBUG:boto:Data: 2023-09-29T22:22:45.179 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.180 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.180 DEBUG:boto:Params: {} 2023-09-29T22:22:45.180 DEBUG:boto:Token: None 2023-09-29T22:22:45.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.180 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk= 2023-09-29T22:22:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk=', 'Content-Length': '0'} 2023-09-29T22:22:45.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eff21aecbfc6961e-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.182 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.183 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.183 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.183 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.183 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.183 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.183 DEBUG:boto:Data: 2023-09-29T22:22:45.183 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.183 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.183 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.183 DEBUG:boto:Params: {} 2023-09-29T22:22:45.184 DEBUG:boto:Token: None 2023-09-29T22:22:45.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mskzMWfAa5NBzHTvIZAtr9VMclw= 2023-09-29T22:22:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mskzMWfAa5NBzHTvIZAtr9VMclw=', 'Content-Length': '0'} 2023-09-29T22:22:45.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007818ab34d6bf5fc0-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.186 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.186 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.186 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.186 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.186 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.186 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.186 DEBUG:boto:Data: 2023-09-29T22:22:45.187 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.187 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.187 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.187 DEBUG:boto:Params: {} 2023-09-29T22:22:45.187 DEBUG:boto:Token: None 2023-09-29T22:22:45.187 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.187 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mskzMWfAa5NBzHTvIZAtr9VMclw= 2023-09-29T22:22:45.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mskzMWfAa5NBzHTvIZAtr9VMclw=', 'Content-Length': '0'} 2023-09-29T22:22:45.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7b83d03c27d26bc-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.190 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.190 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.190 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.190 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.190 DEBUG:boto:Method: GET 2023-09-29T22:22:45.190 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.190 DEBUG:boto:Data: 2023-09-29T22:22:45.190 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.190 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.191 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.191 DEBUG:boto:Params: {} 2023-09-29T22:22:45.191 DEBUG:boto:Token: None 2023-09-29T22:22:45.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.191 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk= 2023-09-29T22:22:45.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk=', 'Content-Length': '0'} 2023-09-29T22:22:45.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef9000ddf015a209-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.234 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.235 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.235 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.235 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.235 DEBUG:boto:Method: GET 2023-09-29T22:22:45.235 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.235 DEBUG:boto:Data: 2023-09-29T22:22:45.235 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.235 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.235 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.235 DEBUG:boto:Params: {} 2023-09-29T22:22:45.236 DEBUG:boto:Token: None 2023-09-29T22:22:45.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:45.236 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk= 2023-09-29T22:22:45.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vXovM/DpljU2Ah1DcbgUJDs8zYk=', 'Content-Length': '0'} 2023-09-29T22:22:45.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c38797b4e7d690b9-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.238 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.238 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.239 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.239 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.239 DEBUG:boto:Method: GET 2023-09-29T22:22:45.239 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.239 DEBUG:boto:Data: 2023-09-29T22:22:45.239 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.239 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.239 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.239 DEBUG:boto:Params: {} 2023-09-29T22:22:45.239 DEBUG:boto:Token: None 2023-09-29T22:22:45.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147a652c0d27195f-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.286 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.287 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.287 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.287 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.287 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.287 DEBUG:boto:Method: GET 2023-09-29T22:22:45.287 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.287 DEBUG:boto:Data: 2023-09-29T22:22:45.287 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.287 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.288 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.288 DEBUG:boto:Params: {} 2023-09-29T22:22:45.288 DEBUG:boto:Token: None 2023-09-29T22:22:45.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efce6e27f40e17ad-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.291 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.291 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.291 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.291 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.291 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.292 DEBUG:boto:Data: 2023-09-29T22:22:45.292 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.292 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.292 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.292 DEBUG:boto:Params: {} 2023-09-29T22:22:45.292 DEBUG:boto:Token: None 2023-09-29T22:22:45.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.296 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d4b0bce5ebc74e3-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.296 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.296 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.296 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.296 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.296 DEBUG:boto:Data: 2023-09-29T22:22:45.296 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.297 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.297 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.297 DEBUG:boto:Params: {} 2023-09-29T22:22:45.297 DEBUG:boto:Token: None 2023-09-29T22:22:45.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.297 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000877dc6c087124baa-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.300 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.300 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.301 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.301 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.301 DEBUG:boto:Method: GET 2023-09-29T22:22:45.301 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.301 DEBUG:boto:Data: 2023-09-29T22:22:45.301 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.301 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.301 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.301 DEBUG:boto:Params: {} 2023-09-29T22:22:45.301 DEBUG:boto:Token: None 2023-09-29T22:22:45.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.302 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad930922ea304c59-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.346 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.346 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.347 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.347 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.347 DEBUG:boto:Method: GET 2023-09-29T22:22:45.347 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.347 DEBUG:boto:Data: 2023-09-29T22:22:45.347 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.347 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.347 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.347 DEBUG:boto:Params: {} 2023-09-29T22:22:45.348 DEBUG:boto:Token: None 2023-09-29T22:22:45.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.348 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224742fd5fe17a96-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.350 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.350 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.350 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.350 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.350 DEBUG:boto:Method: GET 2023-09-29T22:22:45.350 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.350 DEBUG:boto:Data: 2023-09-29T22:22:45.351 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.351 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.351 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.351 DEBUG:boto:Params: {} 2023-09-29T22:22:45.351 DEBUG:boto:Token: None 2023-09-29T22:22:45.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.351 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f2b781d410a3fd6-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.394 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.395 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.395 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.395 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.395 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.395 DEBUG:boto:Method: GET 2023-09-29T22:22:45.395 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.395 DEBUG:boto:Data: 2023-09-29T22:22:45.395 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.396 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.396 DEBUG:boto:Params: {} 2023-09-29T22:22:45.396 DEBUG:boto:Token: None 2023-09-29T22:22:45.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad360136b4523e5-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.399 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.399 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.399 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.399 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.399 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.400 DEBUG:boto:Method: GET 2023-09-29T22:22:45.400 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.400 DEBUG:boto:Data: 2023-09-29T22:22:45.400 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.400 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.400 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.400 DEBUG:boto:Params: {} 2023-09-29T22:22:45.400 DEBUG:boto:Token: None 2023-09-29T22:22:45.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.401 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA= 2023-09-29T22:22:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA=', 'Content-Length': '0'} 2023-09-29T22:22:45.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e201ca5f3f16d66b-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.446 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.447 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.447 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.447 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.447 DEBUG:boto:Method: GET 2023-09-29T22:22:45.447 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.447 DEBUG:boto:Data: 2023-09-29T22:22:45.447 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.447 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.447 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.447 DEBUG:boto:Params: {} 2023-09-29T22:22:45.448 DEBUG:boto:Token: None 2023-09-29T22:22:45.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA= 2023-09-29T22:22:45.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA=', 'Content-Length': '0'} 2023-09-29T22:22:45.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f907fce412bb977b-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.450 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.451 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.451 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.451 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.451 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.451 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.451 DEBUG:boto:Data: 2023-09-29T22:22:45.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.451 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.451 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.452 DEBUG:boto:Params: {} 2023-09-29T22:22:45.452 DEBUG:boto:Token: None 2023-09-29T22:22:45.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1hpoacjR+ZSYuvxlXrsyWNch5So= 2023-09-29T22:22:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1hpoacjR+ZSYuvxlXrsyWNch5So=', 'Content-Length': '0'} 2023-09-29T22:22:45.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfcd99d16728211c-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.454 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.455 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.455 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.455 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.455 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.455 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.455 DEBUG:boto:Data: 2023-09-29T22:22:45.455 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.455 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.455 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.456 DEBUG:boto:Params: {} 2023-09-29T22:22:45.456 DEBUG:boto:Token: None 2023-09-29T22:22:45.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1hpoacjR+ZSYuvxlXrsyWNch5So= 2023-09-29T22:22:45.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1hpoacjR+ZSYuvxlXrsyWNch5So=', 'Content-Length': '0'} 2023-09-29T22:22:45.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2fb5f997e98875c-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.458 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.458 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.458 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.458 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.458 DEBUG:boto:Method: GET 2023-09-29T22:22:45.458 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.459 DEBUG:boto:Data: 2023-09-29T22:22:45.459 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.459 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.459 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.459 DEBUG:boto:Params: {} 2023-09-29T22:22:45.459 DEBUG:boto:Token: None 2023-09-29T22:22:45.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.459 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA= 2023-09-29T22:22:45.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA=', 'Content-Length': '0'} 2023-09-29T22:22:45.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfad6b71a61f36c1-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.502 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.502 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.503 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.503 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.503 DEBUG:boto:Method: GET 2023-09-29T22:22:45.503 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.503 DEBUG:boto:Data: 2023-09-29T22:22:45.503 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.503 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.503 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.503 DEBUG:boto:Params: {} 2023-09-29T22:22:45.503 DEBUG:boto:Token: None 2023-09-29T22:22:45.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:45.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA= 2023-09-29T22:22:45.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A2UhtA3S0rpSMwHHp3zEx6HMeuA=', 'Content-Length': '0'} 2023-09-29T22:22:45.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088bb623f69733ca6-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.506 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.506 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.506 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.506 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.506 DEBUG:boto:Method: GET 2023-09-29T22:22:45.506 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.506 DEBUG:boto:Data: 2023-09-29T22:22:45.506 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.507 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.507 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.507 DEBUG:boto:Params: {} 2023-09-29T22:22:45.507 DEBUG:boto:Token: None 2023-09-29T22:22:45.507 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.507 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e97cb5c851a03b-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.550 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.551 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.551 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.551 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.551 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.551 DEBUG:boto:Method: GET 2023-09-29T22:22:45.551 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.551 DEBUG:boto:Data: 2023-09-29T22:22:45.552 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.552 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.552 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.552 DEBUG:boto:Params: {} 2023-09-29T22:22:45.552 DEBUG:boto:Token: None 2023-09-29T22:22:45.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009664eb999dbe0cad-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.555 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.555 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.556 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.556 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.556 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.556 DEBUG:boto:Data: 2023-09-29T22:22:45.556 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.556 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.556 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.556 DEBUG:boto:Params: {} 2023-09-29T22:22:45.556 DEBUG:boto:Token: None 2023-09-29T22:22:45.557 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.557 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c156246af7a32fc-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.560 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.560 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.560 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.561 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.561 DEBUG:boto:Data: 2023-09-29T22:22:45.561 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.561 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.561 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.561 DEBUG:boto:Params: {} 2023-09-29T22:22:45.561 DEBUG:boto:Token: None 2023-09-29T22:22:45.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.561 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045c69b86cd00168b-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.565 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.565 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.565 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.565 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.565 DEBUG:boto:Method: GET 2023-09-29T22:22:45.565 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.565 DEBUG:boto:Data: 2023-09-29T22:22:45.565 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.566 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.566 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.566 DEBUG:boto:Params: {} 2023-09-29T22:22:45.566 DEBUG:boto:Token: None 2023-09-29T22:22:45.566 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.566 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c24f2a8a2daa41b0-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.610 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.610 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.611 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.611 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.611 DEBUG:boto:Method: GET 2023-09-29T22:22:45.611 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.611 DEBUG:boto:Data: 2023-09-29T22:22:45.611 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.611 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.611 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.611 DEBUG:boto:Params: {} 2023-09-29T22:22:45.612 DEBUG:boto:Token: None 2023-09-29T22:22:45.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.612 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033b6a5896981ee96-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.614 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.614 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.614 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.614 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.615 DEBUG:boto:Method: GET 2023-09-29T22:22:45.615 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.615 DEBUG:boto:Data: 2023-09-29T22:22:45.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.615 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.615 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.615 DEBUG:boto:Params: {} 2023-09-29T22:22:45.615 DEBUG:boto:Token: None 2023-09-29T22:22:45.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.615 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea488396d462462-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.658 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.659 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.659 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.659 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.659 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.659 DEBUG:boto:Method: GET 2023-09-29T22:22:45.659 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.659 DEBUG:boto:Data: 2023-09-29T22:22:45.659 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.659 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.660 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.660 DEBUG:boto:Params: {} 2023-09-29T22:22:45.660 DEBUG:boto:Token: None 2023-09-29T22:22:45.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21e9c1526baa0f6-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.662 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.663 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.663 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.663 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.663 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.663 DEBUG:boto:Method: GET 2023-09-29T22:22:45.663 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.664 DEBUG:boto:Data: 2023-09-29T22:22:45.664 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.664 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.664 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.664 DEBUG:boto:Params: {} 2023-09-29T22:22:45.664 DEBUG:boto:Token: None 2023-09-29T22:22:45.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg= 2023-09-29T22:22:45.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg=', 'Content-Length': '0'} 2023-09-29T22:22:45.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011efd21e67d8ac5d-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.666 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.667 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.667 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.667 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.667 DEBUG:boto:Method: GET 2023-09-29T22:22:45.667 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.667 DEBUG:boto:Data: 2023-09-29T22:22:45.667 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.667 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.667 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.667 DEBUG:boto:Params: {} 2023-09-29T22:22:45.668 DEBUG:boto:Token: None 2023-09-29T22:22:45.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg= 2023-09-29T22:22:45.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg=', 'Content-Length': '0'} 2023-09-29T22:22:45.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009f1878a782d5962-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.670 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.670 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.671 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.671 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.671 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.671 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.671 DEBUG:boto:Data: 2023-09-29T22:22:45.671 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.671 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.671 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.671 DEBUG:boto:Params: {} 2023-09-29T22:22:45.671 DEBUG:boto:Token: None 2023-09-29T22:22:45.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u44Kl4MPVLB1LIlmXMdQbffajBo= 2023-09-29T22:22:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u44Kl4MPVLB1LIlmXMdQbffajBo=', 'Content-Length': '0'} 2023-09-29T22:22:45.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1cc8e4d91f58e77-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.674 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.674 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.674 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.674 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.674 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.675 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.675 DEBUG:boto:Data: 2023-09-29T22:22:45.675 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.675 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.675 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.675 DEBUG:boto:Params: {} 2023-09-29T22:22:45.675 DEBUG:boto:Token: None 2023-09-29T22:22:45.675 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.675 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u44Kl4MPVLB1LIlmXMdQbffajBo= 2023-09-29T22:22:45.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u44Kl4MPVLB1LIlmXMdQbffajBo=', 'Content-Length': '0'} 2023-09-29T22:22:45.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcc6da04d30f69dd-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.678 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.678 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.678 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.678 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.678 DEBUG:boto:Method: GET 2023-09-29T22:22:45.678 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.678 DEBUG:boto:Data: 2023-09-29T22:22:45.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.679 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.679 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.679 DEBUG:boto:Params: {} 2023-09-29T22:22:45.679 DEBUG:boto:Token: None 2023-09-29T22:22:45.679 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.679 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg= 2023-09-29T22:22:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg=', 'Content-Length': '0'} 2023-09-29T22:22:45.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f383de12dee647c-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.682 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.682 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.682 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.682 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.682 DEBUG:boto:Method: GET 2023-09-29T22:22:45.682 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.682 DEBUG:boto:Data: 2023-09-29T22:22:45.682 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.682 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.682 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.683 DEBUG:boto:Params: {} 2023-09-29T22:22:45.683 DEBUG:boto:Token: None 2023-09-29T22:22:45.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:45.683 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg= 2023-09-29T22:22:45.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vrdctW0JeNSyFOEzq11M5FbsfXg=', 'Content-Length': '0'} 2023-09-29T22:22:45.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad19f3e75152b62a-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.685 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.685 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.685 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.685 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.685 DEBUG:boto:Method: GET 2023-09-29T22:22:45.685 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.686 DEBUG:boto:Data: 2023-09-29T22:22:45.686 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.686 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.686 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.686 DEBUG:boto:Params: {} 2023-09-29T22:22:45.686 DEBUG:boto:Token: None 2023-09-29T22:22:45.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f65c208a5f00e9-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.689 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.689 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.690 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.690 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.690 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.690 DEBUG:boto:Method: GET 2023-09-29T22:22:45.690 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.690 DEBUG:boto:Data: 2023-09-29T22:22:45.690 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.690 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.690 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.690 DEBUG:boto:Params: {} 2023-09-29T22:22:45.691 DEBUG:boto:Token: None 2023-09-29T22:22:45.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d033fde0c9f4ec17-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.694 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.694 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.694 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.694 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.694 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.695 DEBUG:boto:Data: 2023-09-29T22:22:45.695 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.695 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.695 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.695 DEBUG:boto:Params: {} 2023-09-29T22:22:45.695 DEBUG:boto:Token: None 2023-09-29T22:22:45.695 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.695 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8f98a2ba9d93fa5-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.698 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.699 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.699 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.699 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:45.699 DEBUG:boto:Data: 2023-09-29T22:22:45.699 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.699 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.699 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.699 DEBUG:boto:Params: {} 2023-09-29T22:22:45.699 DEBUG:boto:Token: None 2023-09-29T22:22:45.700 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj 2023-09-29T22:22:45.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo= 2023-09-29T22:22:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cyuv1j22p1ityS7IlP8Hvdqo2Xo=', 'Content-Length': '0'} 2023-09-29T22:22:45.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01929786781ba37-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.703 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.703 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.703 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.703 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.703 DEBUG:boto:Method: GET 2023-09-29T22:22:45.704 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.704 DEBUG:boto:Data: 2023-09-29T22:22:45.704 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.704 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.704 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.704 DEBUG:boto:Params: {} 2023-09-29T22:22:45.704 DEBUG:boto:Token: None 2023-09-29T22:22:45.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4a049c09c158932-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.746 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.747 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.747 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.747 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.747 DEBUG:boto:Method: GET 2023-09-29T22:22:45.747 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.747 DEBUG:boto:Data: 2023-09-29T22:22:45.747 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.747 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.747 DEBUG:boto:Params: {} 2023-09-29T22:22:45.747 DEBUG:boto:Token: None 2023-09-29T22:22:45.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097f888079b759017-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.750 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.750 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.750 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.750 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.750 DEBUG:boto:Method: GET 2023-09-29T22:22:45.750 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.750 DEBUG:boto:Data: 2023-09-29T22:22:45.750 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.751 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:45.751 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.751 DEBUG:boto:Params: {} 2023-09-29T22:22:45.751 DEBUG:boto:Token: None 2023-09-29T22:22:45.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf9314e84441d9e-0065174e35-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.794 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.795 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.795 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.795 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.795 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.795 DEBUG:boto:Method: GET 2023-09-29T22:22:45.795 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.795 DEBUG:boto:Data: 2023-09-29T22:22:45.795 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.795 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.795 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.796 DEBUG:boto:Params: {} 2023-09-29T22:22:45.796 DEBUG:boto:Token: None 2023-09-29T22:22:45.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.796 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb1b8aad2e243db-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.799 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone2, test-zone1} 2023-09-29T22:22:45.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone3, test-zone1} 2023-09-29T22:22:45.800 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.800 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.800 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.800 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:45.800 DEBUG:boto:Data: 2023-09-29T22:22:45.800 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.800 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.800 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.800 DEBUG:boto:Params: {} 2023-09-29T22:22:45.801 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:22:45.801 DEBUG:boto:Token: None 2023-09-29T22:22:45.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.801 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:leV2Try4mxX/D+XWEH1ElCm1hmQ= 2023-09-29T22:22:45.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:leV2Try4mxX/D+XWEH1ElCm1hmQ=', 'Content-Length': '0'} 2023-09-29T22:22:45.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ea8018a755c0b48-0065174e35-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.805 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.805 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.805 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.806 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:45.806 DEBUG:boto:Data: 2023-09-29T22:22:45.806 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.806 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.806 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.806 DEBUG:boto:Params: {} 2023-09-29T22:22:45.806 DEBUG:boto:Token: None 2023-09-29T22:22:45.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:leV2Try4mxX/D+XWEH1ElCm1hmQ= 2023-09-29T22:22:45.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:leV2Try4mxX/D+XWEH1ElCm1hmQ=', 'Content-Length': '0'} 2023-09-29T22:22:45.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000712ef0ce3ccd0388-0065174e35-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.809 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:22:45.809 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.809 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.809 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.809 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.809 DEBUG:boto:Method: GET 2023-09-29T22:22:45.809 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.810 DEBUG:boto:Data: 2023-09-29T22:22:45.810 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.810 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.810 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.810 DEBUG:boto:Params: {} 2023-09-29T22:22:45.810 DEBUG:boto:Token: None 2023-09-29T22:22:45.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/?versions 2023-09-29T22:22:45.810 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs= 2023-09-29T22:22:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs=', 'Content-Length': '0'} 2023-09-29T22:22:45.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068f7355157bf9052-0065174e35-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.814 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.815 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:45.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:22:45.818 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.818 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.819 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.819 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.819 DEBUG:boto:Method: GET 2023-09-29T22:22:45.819 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.819 DEBUG:boto:Data: 2023-09-29T22:22:45.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.819 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.819 DEBUG:boto:Params: {} 2023-09-29T22:22:45.819 DEBUG:boto:Token: None 2023-09-29T22:22:45.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/?versions 2023-09-29T22:22:45.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs= 2023-09-29T22:22:45.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs=', 'Content-Length': '0'} 2023-09-29T22:22:45.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e339f91e594994d-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.823 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:45.828 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.828 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.828 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.828 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.828 DEBUG:boto:Method: GET 2023-09-29T22:22:45.828 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.828 DEBUG:boto:Data: 2023-09-29T22:22:45.828 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.828 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.828 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.829 DEBUG:boto:Params: {} 2023-09-29T22:22:45.829 DEBUG:boto:Token: None 2023-09-29T22:22:45.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/?versions 2023-09-29T22:22:45.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs= 2023-09-29T22:22:45.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs=', 'Content-Length': '0'} 2023-09-29T22:22:45.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbcdd6bbbe65663c-0065174e35-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.832 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.833 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.834 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.834 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.834 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.834 DEBUG:boto:Method: GET 2023-09-29T22:22:45.834 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:45.834 DEBUG:boto:Data: 2023-09-29T22:22:45.834 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.834 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.834 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.835 DEBUG:boto:Params: {} 2023-09-29T22:22:45.835 DEBUG:boto:Token: None 2023-09-29T22:22:45.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/?versions 2023-09-29T22:22:45.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs= 2023-09-29T22:22:45.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Zbn+XCKMCvLTmoaUq32BPSs3fs=', 'Content-Length': '0'} 2023-09-29T22:22:45.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000358c1f402e7fc45e-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.839 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:45.840 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.841 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.841 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.841 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.841 DEBUG:boto:Method: GET 2023-09-29T22:22:45.841 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.841 DEBUG:boto:Data: 2023-09-29T22:22:45.841 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.841 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.841 DEBUG:boto:Params: {} 2023-09-29T22:22:45.842 DEBUG:boto:Token: None 2023-09-29T22:22:45.842 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000148671f6bb2ead1a-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.844 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.844 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.844 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.844 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.845 DEBUG:boto:Method: GET 2023-09-29T22:22:45.845 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.845 DEBUG:boto:Data: 2023-09-29T22:22:45.845 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.845 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.845 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.845 DEBUG:boto:Params: {} 2023-09-29T22:22:45.845 DEBUG:boto:Token: None 2023-09-29T22:22:45.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d8e78cdd300a51a-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.847 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.848 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.848 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.848 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.848 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.848 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.848 DEBUG:boto:Data: 2023-09-29T22:22:45.848 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.848 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.848 DEBUG:boto:Params: {} 2023-09-29T22:22:45.849 DEBUG:boto:Token: None 2023-09-29T22:22:45.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.849 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bpsq89rEV54TOyFVn5gmaopT4S4= 2023-09-29T22:22:45.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bpsq89rEV54TOyFVn5gmaopT4S4=', 'Content-Length': '0'} 2023-09-29T22:22:45.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066dcd613bd638de3-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.851 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.851 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.851 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.851 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.851 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.851 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.852 DEBUG:boto:Data: 2023-09-29T22:22:45.852 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.852 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.852 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.852 DEBUG:boto:Params: {} 2023-09-29T22:22:45.852 DEBUG:boto:Token: None 2023-09-29T22:22:45.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Bpsq89rEV54TOyFVn5gmaopT4S4= 2023-09-29T22:22:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Bpsq89rEV54TOyFVn5gmaopT4S4=', 'Content-Length': '0'} 2023-09-29T22:22:45.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e60dc87d0d97a93e-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:45.854 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.855 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.855 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.855 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.855 DEBUG:boto:Method: GET 2023-09-29T22:22:45.855 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.855 DEBUG:boto:Data: 2023-09-29T22:22:45.855 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.855 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.855 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.855 DEBUG:boto:Params: {} 2023-09-29T22:22:45.856 DEBUG:boto:Token: None 2023-09-29T22:22:45.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dff0e8ba02be2b9b-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.858 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.858 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.858 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.858 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.858 DEBUG:boto:Method: GET 2023-09-29T22:22:45.858 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.858 DEBUG:boto:Data: 2023-09-29T22:22:45.858 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.859 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.859 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.859 DEBUG:boto:Params: {} 2023-09-29T22:22:45.859 DEBUG:boto:Token: None 2023-09-29T22:22:45.859 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.859 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000558907f358af4538-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.861 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.861 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.861 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.861 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.861 DEBUG:boto:Method: GET 2023-09-29T22:22:45.862 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.862 DEBUG:boto:Data: 2023-09-29T22:22:45.862 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.862 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.862 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.862 DEBUG:boto:Params: {} 2023-09-29T22:22:45.862 DEBUG:boto:Token: None 2023-09-29T22:22:45.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.862 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000608840fd20a5eb64-0065174e35-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.865 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.865 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.865 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.865 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.866 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.866 DEBUG:boto:Method: GET 2023-09-29T22:22:45.866 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.866 DEBUG:boto:Data: 2023-09-29T22:22:45.866 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.866 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.866 DEBUG:boto:Params: {} 2023-09-29T22:22:45.866 DEBUG:boto:Token: None 2023-09-29T22:22:45.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093587d9b5fdd4421-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.910 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.911 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.911 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.911 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.911 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:45.911 DEBUG:boto:Data: 2023-09-29T22:22:45.911 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.912 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.912 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.912 DEBUG:boto:Params: {} 2023-09-29T22:22:45.912 DEBUG:boto:Token: None 2023-09-29T22:22:45.912 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy 2023-09-29T22:22:45.912 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y+davKlNdS0oD6LiKs++z0ioatI= 2023-09-29T22:22:45.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y+davKlNdS0oD6LiKs++z0ioatI=', 'Content-Length': '0'} 2023-09-29T22:22:45.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2228d2fcc3f75d9-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.915 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.915 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.915 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.915 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:45.915 DEBUG:boto:Data: 2023-09-29T22:22:45.915 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.915 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.915 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.916 DEBUG:boto:Params: {} 2023-09-29T22:22:45.916 DEBUG:boto:Token: None 2023-09-29T22:22:45.916 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy 2023-09-29T22:22:45.916 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y+davKlNdS0oD6LiKs++z0ioatI= 2023-09-29T22:22:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y+davKlNdS0oD6LiKs++z0ioatI=', 'Content-Length': '0'} 2023-09-29T22:22:45.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000756c1e18c4a93eea-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:45.919 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.919 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.919 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.919 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.919 DEBUG:boto:Method: GET 2023-09-29T22:22:45.919 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.919 DEBUG:boto:Data: 2023-09-29T22:22:45.919 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.920 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.920 DEBUG:boto:Params: {} 2023-09-29T22:22:45.920 DEBUG:boto:Token: None 2023-09-29T22:22:45.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c37a877634bfe31b-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.923 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:45.923 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:45.923 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.923 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.923 DEBUG:boto:Method: GET 2023-09-29T22:22:45.923 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.923 DEBUG:boto:Data: 2023-09-29T22:22:45.924 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.924 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.924 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.924 DEBUG:boto:Params: {} 2023-09-29T22:22:45.924 DEBUG:boto:Token: None 2023-09-29T22:22:45.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:45.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI= 2023-09-29T22:22:45.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PKGkEJ3GckDd0QtSsRXyDARfRXI=', 'Content-Length': '0'} 2023-09-29T22:22:45.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc725d479ecdef5d-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.926 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.926 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.926 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.926 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.927 DEBUG:boto:Method: GET 2023-09-29T22:22:45.927 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.927 DEBUG:boto:Data: 2023-09-29T22:22:45.927 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.927 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.927 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.927 DEBUG:boto:Params: {} 2023-09-29T22:22:45.927 DEBUG:boto:Token: None 2023-09-29T22:22:45.927 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d174895b2dfcaca3-0065174e35-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.970 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.971 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.971 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.971 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.971 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.971 DEBUG:boto:Method: GET 2023-09-29T22:22:45.971 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:45.971 DEBUG:boto:Data: 2023-09-29T22:22:45.971 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.971 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.972 DEBUG:boto:Params: {} 2023-09-29T22:22:45.972 DEBUG:boto:Token: None 2023-09-29T22:22:45.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:45.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005afb03c8bf805705-0065174e35-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.974 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:45.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.975 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.975 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.975 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.975 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.975 DEBUG:boto:Method: GET 2023-09-29T22:22:45.976 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.976 DEBUG:boto:Data: 2023-09-29T22:22:45.976 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.976 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.976 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.976 DEBUG:boto:Params: {} 2023-09-29T22:22:45.976 DEBUG:boto:Token: None 2023-09-29T22:22:45.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.976 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da2660573268100c-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.978 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.979 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.979 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.979 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.979 DEBUG:boto:Method: GET 2023-09-29T22:22:45.979 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.979 DEBUG:boto:Data: 2023-09-29T22:22:45.979 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.979 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.979 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.980 DEBUG:boto:Params: {} 2023-09-29T22:22:45.980 DEBUG:boto:Token: None 2023-09-29T22:22:45.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.980 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005af73298ca5a4f21-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.982 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.982 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.982 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.982 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.982 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.983 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.983 DEBUG:boto:Data: 2023-09-29T22:22:45.983 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.983 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.983 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.983 DEBUG:boto:Params: {} 2023-09-29T22:22:45.983 DEBUG:boto:Token: None 2023-09-29T22:22:45.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.983 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kBUVaekJwCJ7CamgPbvFEl4G1Ls= 2023-09-29T22:22:45.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kBUVaekJwCJ7CamgPbvFEl4G1Ls=', 'Content-Length': '0'} 2023-09-29T22:22:45.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d1d821c641d876f-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.985 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.986 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.986 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.986 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.986 DEBUG:boto:Method: HEAD 2023-09-29T22:22:45.986 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.986 DEBUG:boto:Data: 2023-09-29T22:22:45.986 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.986 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.986 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.987 DEBUG:boto:Params: {} 2023-09-29T22:22:45.987 DEBUG:boto:Token: None 2023-09-29T22:22:45.987 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kBUVaekJwCJ7CamgPbvFEl4G1Ls= 2023-09-29T22:22:45.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kBUVaekJwCJ7CamgPbvFEl4G1Ls=', 'Content-Length': '0'} 2023-09-29T22:22:45.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043548b04b994c875-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:45.989 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.989 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.989 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.989 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.990 DEBUG:boto:Method: GET 2023-09-29T22:22:45.990 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.990 DEBUG:boto:Data: 2023-09-29T22:22:45.990 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.990 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.990 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.990 DEBUG:boto:Params: {} 2023-09-29T22:22:45.990 DEBUG:boto:Token: None 2023-09-29T22:22:45.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.991 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008f5f78a74c5b860-0065174e35-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.993 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:45.993 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:45.993 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.993 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.993 DEBUG:boto:Method: GET 2023-09-29T22:22:45.993 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.993 DEBUG:boto:Data: 2023-09-29T22:22:45.993 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.993 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:45.993 DEBUG:boto:Port: 8000 2023-09-29T22:22:45.994 DEBUG:boto:Params: {} 2023-09-29T22:22:45.994 DEBUG:boto:Token: None 2023-09-29T22:22:45.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:45.994 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA= 2023-09-29T22:22:45.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qkvH7FDzjxmPVz2gdPNB84d+GgA=', 'Content-Length': '0'} 2023-09-29T22:22:45.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000087ed4dedb97ae2d-0065174e35-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:45.996 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:45.996 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:45.996 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.996 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.996 DEBUG:boto:Method: GET 2023-09-29T22:22:45.996 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:45.997 DEBUG:boto:Data: 2023-09-29T22:22:45.997 DEBUG:boto:Headers: {} 2023-09-29T22:22:45.997 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:45.997 DEBUG:boto:Port: 8001 2023-09-29T22:22:45.997 DEBUG:boto:Params: {} 2023-09-29T22:22:45.997 DEBUG:boto:Token: None 2023-09-29T22:22:45.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:45 GMT /dmuhsu-19/ 2023-09-29T22:22:45.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU= 2023-09-29T22:22:45.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oR1Ry+UBl++/psSHHMK7VlKJptU=', 'Content-Length': '0'} 2023-09-29T22:22:46.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047337255e8625e7d-0065174e35-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.000 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.001 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.001 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.001 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.001 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.001 DEBUG:boto:Method: GET 2023-09-29T22:22:46.001 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.001 DEBUG:boto:Data: 2023-09-29T22:22:46.001 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.001 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.001 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.002 DEBUG:boto:Params: {} 2023-09-29T22:22:46.002 DEBUG:boto:Token: None 2023-09-29T22:22:46.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.002 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005674aa6530ec911e-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.005 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.005 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.005 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.005 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.005 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.005 DEBUG:boto:Data: 2023-09-29T22:22:46.005 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.006 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.006 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.006 DEBUG:boto:Params: {} 2023-09-29T22:22:46.006 DEBUG:boto:Token: None 2023-09-29T22:22:46.006 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.006 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d757ba7c23e2d6f-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.009 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.010 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.010 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.010 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.010 DEBUG:boto:Data: 2023-09-29T22:22:46.010 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.010 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.010 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.010 DEBUG:boto:Params: {} 2023-09-29T22:22:46.010 DEBUG:boto:Token: None 2023-09-29T22:22:46.011 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e2392c5aa6588dd-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.014 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.014 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.014 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.014 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.015 DEBUG:boto:Method: GET 2023-09-29T22:22:46.015 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.015 DEBUG:boto:Data: 2023-09-29T22:22:46.015 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.015 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.015 DEBUG:boto:Params: {} 2023-09-29T22:22:46.015 DEBUG:boto:Token: None 2023-09-29T22:22:46.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2f87497bf7ef53d-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.058 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.058 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.059 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.059 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.059 DEBUG:boto:Method: GET 2023-09-29T22:22:46.059 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.059 DEBUG:boto:Data: 2023-09-29T22:22:46.059 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.059 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.059 DEBUG:boto:Params: {} 2023-09-29T22:22:46.060 DEBUG:boto:Token: None 2023-09-29T22:22:46.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bd31dfb53602474-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.062 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.062 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.062 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.062 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.063 DEBUG:boto:Method: GET 2023-09-29T22:22:46.063 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.063 DEBUG:boto:Data: 2023-09-29T22:22:46.063 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.063 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.063 DEBUG:boto:Params: {} 2023-09-29T22:22:46.063 DEBUG:boto:Token: None 2023-09-29T22:22:46.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eb37f7a9a1ed9a3-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.067 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.067 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.067 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.067 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.067 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.067 DEBUG:boto:Method: GET 2023-09-29T22:22:46.067 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.067 DEBUG:boto:Data: 2023-09-29T22:22:46.068 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.068 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.068 DEBUG:boto:Params: {} 2023-09-29T22:22:46.068 DEBUG:boto:Token: None 2023-09-29T22:22:46.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c07fcf660c167b6d-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.071 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.071 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.071 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.071 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.071 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.072 DEBUG:boto:Method: GET 2023-09-29T22:22:46.072 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.072 DEBUG:boto:Data: 2023-09-29T22:22:46.072 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.072 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.072 DEBUG:boto:Params: {} 2023-09-29T22:22:46.072 DEBUG:boto:Token: None 2023-09-29T22:22:46.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w= 2023-09-29T22:22:46.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w=', 'Content-Length': '0'} 2023-09-29T22:22:46.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aab178d52824a8d8-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.074 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.074 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.075 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.075 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.075 DEBUG:boto:Method: GET 2023-09-29T22:22:46.075 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.075 DEBUG:boto:Data: 2023-09-29T22:22:46.075 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.075 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.075 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.075 DEBUG:boto:Params: {} 2023-09-29T22:22:46.075 DEBUG:boto:Token: None 2023-09-29T22:22:46.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w= 2023-09-29T22:22:46.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w=', 'Content-Length': '0'} 2023-09-29T22:22:46.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eb0bcef6fc6e583-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.118 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.118 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.119 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.119 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.119 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.119 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.119 DEBUG:boto:Data: 2023-09-29T22:22:46.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.119 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.119 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.119 DEBUG:boto:Params: {} 2023-09-29T22:22:46.119 DEBUG:boto:Token: None 2023-09-29T22:22:46.120 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FiDiRSH8zHJlRg2qj6kpbXOZ2+o= 2023-09-29T22:22:46.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FiDiRSH8zHJlRg2qj6kpbXOZ2+o=', 'Content-Length': '0'} 2023-09-29T22:22:46.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab347fd25a986726-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.123 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.123 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.123 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.123 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.124 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.124 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.124 DEBUG:boto:Data: 2023-09-29T22:22:46.124 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.124 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.124 DEBUG:boto:Params: {} 2023-09-29T22:22:46.124 DEBUG:boto:Token: None 2023-09-29T22:22:46.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.125 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FiDiRSH8zHJlRg2qj6kpbXOZ2+o= 2023-09-29T22:22:46.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FiDiRSH8zHJlRg2qj6kpbXOZ2+o=', 'Content-Length': '0'} 2023-09-29T22:22:46.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c52ef714080ba6c-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.126 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.127 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.127 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.127 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.127 DEBUG:boto:Method: GET 2023-09-29T22:22:46.127 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.127 DEBUG:boto:Data: 2023-09-29T22:22:46.127 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.127 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.127 DEBUG:boto:Params: {} 2023-09-29T22:22:46.128 DEBUG:boto:Token: None 2023-09-29T22:22:46.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w= 2023-09-29T22:22:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w=', 'Content-Length': '0'} 2023-09-29T22:22:46.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070e2183653c09f49-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.174 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.175 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.175 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.175 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.175 DEBUG:boto:Method: GET 2023-09-29T22:22:46.175 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.175 DEBUG:boto:Data: 2023-09-29T22:22:46.175 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.175 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.175 DEBUG:boto:Params: {} 2023-09-29T22:22:46.176 DEBUG:boto:Token: None 2023-09-29T22:22:46.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:46.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w= 2023-09-29T22:22:46.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:US3TDeUGWi0OEXHCoF6zfSmyH/w=', 'Content-Length': '0'} 2023-09-29T22:22:46.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072b5f223c3822850-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.218 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.218 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.219 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.219 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.219 DEBUG:boto:Method: GET 2023-09-29T22:22:46.219 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.219 DEBUG:boto:Data: 2023-09-29T22:22:46.219 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.219 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.219 DEBUG:boto:Params: {} 2023-09-29T22:22:46.219 DEBUG:boto:Token: None 2023-09-29T22:22:46.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b941a8d628e84e-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.222 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.223 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.223 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.223 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.223 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.223 DEBUG:boto:Method: GET 2023-09-29T22:22:46.223 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.223 DEBUG:boto:Data: 2023-09-29T22:22:46.223 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.224 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.224 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.224 DEBUG:boto:Params: {} 2023-09-29T22:22:46.224 DEBUG:boto:Token: None 2023-09-29T22:22:46.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6daae1f02539ed5-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.227 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.227 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.227 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.227 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.227 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.228 DEBUG:boto:Data: 2023-09-29T22:22:46.228 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.228 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.228 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.228 DEBUG:boto:Params: {} 2023-09-29T22:22:46.228 DEBUG:boto:Token: None 2023-09-29T22:22:46.228 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005172e0f954560ec3-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.231 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.231 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.231 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.231 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.231 DEBUG:boto:Data: 2023-09-29T22:22:46.232 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.232 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.232 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.232 DEBUG:boto:Params: {} 2023-09-29T22:22:46.232 DEBUG:boto:Token: None 2023-09-29T22:22:46.232 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c203bd3bfef3631-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.235 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.236 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.236 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.236 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.236 DEBUG:boto:Method: GET 2023-09-29T22:22:46.236 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.236 DEBUG:boto:Data: 2023-09-29T22:22:46.236 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.236 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.236 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.236 DEBUG:boto:Params: {} 2023-09-29T22:22:46.237 DEBUG:boto:Token: None 2023-09-29T22:22:46.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e9fa80e0d286df1-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.239 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.239 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.239 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.239 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.239 DEBUG:boto:Method: GET 2023-09-29T22:22:46.239 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.239 DEBUG:boto:Data: 2023-09-29T22:22:46.240 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.240 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.240 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.240 DEBUG:boto:Params: {} 2023-09-29T22:22:46.240 DEBUG:boto:Token: None 2023-09-29T22:22:46.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a924cedcba815bec-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.282 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.283 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.283 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.283 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.283 DEBUG:boto:Method: GET 2023-09-29T22:22:46.283 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.283 DEBUG:boto:Data: 2023-09-29T22:22:46.283 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.283 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.283 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.283 DEBUG:boto:Params: {} 2023-09-29T22:22:46.284 DEBUG:boto:Token: None 2023-09-29T22:22:46.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e56b13849bf42f42-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.286 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.287 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.287 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.287 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.287 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.287 DEBUG:boto:Method: GET 2023-09-29T22:22:46.287 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.287 DEBUG:boto:Data: 2023-09-29T22:22:46.288 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.288 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.288 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.288 DEBUG:boto:Params: {} 2023-09-29T22:22:46.288 DEBUG:boto:Token: None 2023-09-29T22:22:46.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fbb6626d1734789-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.291 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.291 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.292 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.292 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.292 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.292 DEBUG:boto:Method: GET 2023-09-29T22:22:46.292 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.292 DEBUG:boto:Data: 2023-09-29T22:22:46.292 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.292 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.292 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.292 DEBUG:boto:Params: {} 2023-09-29T22:22:46.293 DEBUG:boto:Token: None 2023-09-29T22:22:46.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.293 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0= 2023-09-29T22:22:46.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0=', 'Content-Length': '0'} 2023-09-29T22:22:46.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011c074391d1b1755-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.295 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.295 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.295 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.295 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.295 DEBUG:boto:Method: GET 2023-09-29T22:22:46.295 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.296 DEBUG:boto:Data: 2023-09-29T22:22:46.296 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.296 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.296 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.296 DEBUG:boto:Params: {} 2023-09-29T22:22:46.296 DEBUG:boto:Token: None 2023-09-29T22:22:46.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0= 2023-09-29T22:22:46.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0=', 'Content-Length': '0'} 2023-09-29T22:22:46.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e57930e95a7df8-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.298 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.298 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.298 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.298 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.299 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.299 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.299 DEBUG:boto:Data: 2023-09-29T22:22:46.299 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.299 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.299 DEBUG:boto:Params: {} 2023-09-29T22:22:46.299 DEBUG:boto:Token: None 2023-09-29T22:22:46.299 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vQyel89a2fRe/MYzcVtImVwP+gw= 2023-09-29T22:22:46.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vQyel89a2fRe/MYzcVtImVwP+gw=', 'Content-Length': '0'} 2023-09-29T22:22:46.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d331802560e5a475-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.301 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.301 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.302 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.302 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.302 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.302 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.302 DEBUG:boto:Data: 2023-09-29T22:22:46.302 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.302 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.302 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.302 DEBUG:boto:Params: {} 2023-09-29T22:22:46.302 DEBUG:boto:Token: None 2023-09-29T22:22:46.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vQyel89a2fRe/MYzcVtImVwP+gw= 2023-09-29T22:22:46.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vQyel89a2fRe/MYzcVtImVwP+gw=', 'Content-Length': '0'} 2023-09-29T22:22:46.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe007d79ac89e71-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.305 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.305 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.305 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.305 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.305 DEBUG:boto:Method: GET 2023-09-29T22:22:46.305 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.305 DEBUG:boto:Data: 2023-09-29T22:22:46.305 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.306 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.306 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.306 DEBUG:boto:Params: {} 2023-09-29T22:22:46.306 DEBUG:boto:Token: None 2023-09-29T22:22:46.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.306 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0= 2023-09-29T22:22:46.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0=', 'Content-Length': '0'} 2023-09-29T22:22:46.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f657b1f815924a5-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.350 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.350 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.351 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.351 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.351 DEBUG:boto:Method: GET 2023-09-29T22:22:46.351 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.351 DEBUG:boto:Data: 2023-09-29T22:22:46.351 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.351 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.351 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.351 DEBUG:boto:Params: {} 2023-09-29T22:22:46.351 DEBUG:boto:Token: None 2023-09-29T22:22:46.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:46.352 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0= 2023-09-29T22:22:46.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mnHwYmV3YvgG+s1g5k7gkxS0+e0=', 'Content-Length': '0'} 2023-09-29T22:22:46.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d85eb26800ae387d-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.394 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.394 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.395 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.395 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.395 DEBUG:boto:Method: GET 2023-09-29T22:22:46.395 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.395 DEBUG:boto:Data: 2023-09-29T22:22:46.395 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.395 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.395 DEBUG:boto:Params: {} 2023-09-29T22:22:46.395 DEBUG:boto:Token: None 2023-09-29T22:22:46.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002435fc7597cbc2e7-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.398 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.399 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.399 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.399 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.399 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.399 DEBUG:boto:Method: GET 2023-09-29T22:22:46.399 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.399 DEBUG:boto:Data: 2023-09-29T22:22:46.399 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.400 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.400 DEBUG:boto:Params: {} 2023-09-29T22:22:46.400 DEBUG:boto:Token: None 2023-09-29T22:22:46.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096e26561775498fe-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.446 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.447 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.447 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.447 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.447 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.447 DEBUG:boto:Data: 2023-09-29T22:22:46.447 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.447 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.447 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.447 DEBUG:boto:Params: {} 2023-09-29T22:22:46.447 DEBUG:boto:Token: None 2023-09-29T22:22:46.448 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd5bfc834ed21e87-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.451 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.451 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.451 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.451 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.451 DEBUG:boto:Data: 2023-09-29T22:22:46.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.451 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.452 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.452 DEBUG:boto:Params: {} 2023-09-29T22:22:46.452 DEBUG:boto:Token: None 2023-09-29T22:22:46.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000920d57e08bd5a9bf-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.455 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.455 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.455 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.455 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.456 DEBUG:boto:Method: GET 2023-09-29T22:22:46.456 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.456 DEBUG:boto:Data: 2023-09-29T22:22:46.456 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.456 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.456 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.456 DEBUG:boto:Params: {} 2023-09-29T22:22:46.456 DEBUG:boto:Token: None 2023-09-29T22:22:46.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.457 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014053732f4c163ed-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.459 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.459 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.459 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.459 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.459 DEBUG:boto:Method: GET 2023-09-29T22:22:46.459 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.459 DEBUG:boto:Data: 2023-09-29T22:22:46.459 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.460 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.460 DEBUG:boto:Params: {} 2023-09-29T22:22:46.460 DEBUG:boto:Token: None 2023-09-29T22:22:46.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1548ab2e56d285c-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.462 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.462 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.462 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.463 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.463 DEBUG:boto:Method: GET 2023-09-29T22:22:46.463 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.463 DEBUG:boto:Data: 2023-09-29T22:22:46.463 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.463 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.463 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.463 DEBUG:boto:Params: {} 2023-09-29T22:22:46.463 DEBUG:boto:Token: None 2023-09-29T22:22:46.463 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df95c4dc4530621-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.466 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.467 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.467 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.467 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.467 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.467 DEBUG:boto:Method: GET 2023-09-29T22:22:46.467 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.467 DEBUG:boto:Data: 2023-09-29T22:22:46.468 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.468 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.468 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.468 DEBUG:boto:Params: {} 2023-09-29T22:22:46.468 DEBUG:boto:Token: None 2023-09-29T22:22:46.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.468 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc841012926a33a5-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.471 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.471 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.471 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.471 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.471 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.472 DEBUG:boto:Method: GET 2023-09-29T22:22:46.472 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.472 DEBUG:boto:Data: 2023-09-29T22:22:46.472 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.472 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.472 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.472 DEBUG:boto:Params: {} 2023-09-29T22:22:46.472 DEBUG:boto:Token: None 2023-09-29T22:22:46.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs= 2023-09-29T22:22:46.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs=', 'Content-Length': '0'} 2023-09-29T22:22:46.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aa304731e66c0dc-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.514 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.514 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.515 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.515 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.515 DEBUG:boto:Method: GET 2023-09-29T22:22:46.515 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.515 DEBUG:boto:Data: 2023-09-29T22:22:46.515 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.515 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.515 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.515 DEBUG:boto:Params: {} 2023-09-29T22:22:46.515 DEBUG:boto:Token: None 2023-09-29T22:22:46.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs= 2023-09-29T22:22:46.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs=', 'Content-Length': '0'} 2023-09-29T22:22:46.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000764a88af557f12cf-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.558 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.559 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.559 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.559 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.559 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.559 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.559 DEBUG:boto:Data: 2023-09-29T22:22:46.559 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.559 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.559 DEBUG:boto:Params: {} 2023-09-29T22:22:46.560 DEBUG:boto:Token: None 2023-09-29T22:22:46.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v7kP1rRWd06+5b7QZnlYibm6qVg= 2023-09-29T22:22:46.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v7kP1rRWd06+5b7QZnlYibm6qVg=', 'Content-Length': '0'} 2023-09-29T22:22:46.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ed55eed99419c13-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.562 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.562 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.562 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.562 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.562 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.563 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.563 DEBUG:boto:Data: 2023-09-29T22:22:46.563 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.563 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.563 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.563 DEBUG:boto:Params: {} 2023-09-29T22:22:46.563 DEBUG:boto:Token: None 2023-09-29T22:22:46.563 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v7kP1rRWd06+5b7QZnlYibm6qVg= 2023-09-29T22:22:46.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v7kP1rRWd06+5b7QZnlYibm6qVg=', 'Content-Length': '0'} 2023-09-29T22:22:46.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2b7e04480ed49ee-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.566 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.566 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.566 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.566 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.566 DEBUG:boto:Method: GET 2023-09-29T22:22:46.566 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.566 DEBUG:boto:Data: 2023-09-29T22:22:46.566 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.566 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.567 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.567 DEBUG:boto:Params: {} 2023-09-29T22:22:46.567 DEBUG:boto:Token: None 2023-09-29T22:22:46.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.567 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs= 2023-09-29T22:22:46.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs=', 'Content-Length': '0'} 2023-09-29T22:22:46.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e9dca0f3802c872-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.569 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.569 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.569 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.569 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.569 DEBUG:boto:Method: GET 2023-09-29T22:22:46.569 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.570 DEBUG:boto:Data: 2023-09-29T22:22:46.570 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.570 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.570 DEBUG:boto:Params: {} 2023-09-29T22:22:46.570 DEBUG:boto:Token: None 2023-09-29T22:22:46.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:46.570 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs= 2023-09-29T22:22:46.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7e/Hir1jkDhGFTR5SJ8fxIDBuNs=', 'Content-Length': '0'} 2023-09-29T22:22:46.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba20117eb63bd7d1-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.614 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.615 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.615 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.615 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.615 DEBUG:boto:Method: GET 2023-09-29T22:22:46.615 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.615 DEBUG:boto:Data: 2023-09-29T22:22:46.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.615 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.615 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.616 DEBUG:boto:Params: {} 2023-09-29T22:22:46.616 DEBUG:boto:Token: None 2023-09-29T22:22:46.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096f00be1f77a7bb1-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.619 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.619 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.619 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.619 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.619 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.619 DEBUG:boto:Method: GET 2023-09-29T22:22:46.620 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.620 DEBUG:boto:Data: 2023-09-29T22:22:46.620 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.620 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.620 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.620 DEBUG:boto:Params: {} 2023-09-29T22:22:46.620 DEBUG:boto:Token: None 2023-09-29T22:22:46.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000252649ee54e6d232-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.623 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.623 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.623 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.623 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.623 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.623 DEBUG:boto:Data: 2023-09-29T22:22:46.623 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.624 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.624 DEBUG:boto:Params: {} 2023-09-29T22:22:46.624 DEBUG:boto:Token: None 2023-09-29T22:22:46.624 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030471f34d987d923-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.627 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.627 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.627 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.627 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.627 DEBUG:boto:Data: 2023-09-29T22:22:46.627 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.627 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.627 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.628 DEBUG:boto:Params: {} 2023-09-29T22:22:46.628 DEBUG:boto:Token: None 2023-09-29T22:22:46.628 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.628 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da7c7275568ea710-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.631 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.631 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.631 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.631 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.631 DEBUG:boto:Method: GET 2023-09-29T22:22:46.631 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.631 DEBUG:boto:Data: 2023-09-29T22:22:46.631 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.631 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.632 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.632 DEBUG:boto:Params: {} 2023-09-29T22:22:46.632 DEBUG:boto:Token: None 2023-09-29T22:22:46.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.632 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ab92ff57d14e7b6-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.634 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.634 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.634 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.634 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.634 DEBUG:boto:Method: GET 2023-09-29T22:22:46.634 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.635 DEBUG:boto:Data: 2023-09-29T22:22:46.635 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.635 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.635 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.635 DEBUG:boto:Params: {} 2023-09-29T22:22:46.635 DEBUG:boto:Token: None 2023-09-29T22:22:46.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.635 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087e04319892bc8c9-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.678 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.679 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.679 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.679 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.679 DEBUG:boto:Method: GET 2023-09-29T22:22:46.679 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.679 DEBUG:boto:Data: 2023-09-29T22:22:46.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.679 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.679 DEBUG:boto:Params: {} 2023-09-29T22:22:46.680 DEBUG:boto:Token: None 2023-09-29T22:22:46.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab45a0f023918f8-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.682 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.683 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.683 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.683 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.683 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.683 DEBUG:boto:Method: GET 2023-09-29T22:22:46.683 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.684 DEBUG:boto:Data: 2023-09-29T22:22:46.684 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.684 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.684 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.684 DEBUG:boto:Params: {} 2023-09-29T22:22:46.684 DEBUG:boto:Token: None 2023-09-29T22:22:46.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be57de867c47f6cf-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.686 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.687 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.687 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.687 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.687 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.687 DEBUG:boto:Method: GET 2023-09-29T22:22:46.688 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.688 DEBUG:boto:Data: 2023-09-29T22:22:46.688 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.688 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.688 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.688 DEBUG:boto:Params: {} 2023-09-29T22:22:46.688 DEBUG:boto:Token: None 2023-09-29T22:22:46.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.688 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4= 2023-09-29T22:22:46.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4=', 'Content-Length': '0'} 2023-09-29T22:22:46.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000717f721c063d0cc1-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.730 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.730 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.731 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.731 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.731 DEBUG:boto:Method: GET 2023-09-29T22:22:46.731 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.731 DEBUG:boto:Data: 2023-09-29T22:22:46.731 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.731 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.731 DEBUG:boto:Params: {} 2023-09-29T22:22:46.732 DEBUG:boto:Token: None 2023-09-29T22:22:46.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4= 2023-09-29T22:22:46.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4=', 'Content-Length': '0'} 2023-09-29T22:22:46.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b78793b56dd3dff-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.734 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.734 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.734 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.734 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.734 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.734 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.734 DEBUG:boto:Data: 2023-09-29T22:22:46.734 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.734 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.735 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.735 DEBUG:boto:Params: {} 2023-09-29T22:22:46.735 DEBUG:boto:Token: None 2023-09-29T22:22:46.735 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.735 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zq+Yd5zZ+vnXEOko7eMx3n0e/DE= 2023-09-29T22:22:46.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zq+Yd5zZ+vnXEOko7eMx3n0e/DE=', 'Content-Length': '0'} 2023-09-29T22:22:46.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3ba99e5f0b3f407-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.737 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.737 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.738 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.738 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.738 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.738 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.738 DEBUG:boto:Data: 2023-09-29T22:22:46.738 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.738 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.738 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.738 DEBUG:boto:Params: {} 2023-09-29T22:22:46.738 DEBUG:boto:Token: None 2023-09-29T22:22:46.739 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.739 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zq+Yd5zZ+vnXEOko7eMx3n0e/DE= 2023-09-29T22:22:46.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zq+Yd5zZ+vnXEOko7eMx3n0e/DE=', 'Content-Length': '0'} 2023-09-29T22:22:46.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e477aea14904a77-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.741 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.741 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.741 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.741 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.741 DEBUG:boto:Method: GET 2023-09-29T22:22:46.741 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.742 DEBUG:boto:Data: 2023-09-29T22:22:46.742 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.742 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.742 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.742 DEBUG:boto:Params: {} 2023-09-29T22:22:46.742 DEBUG:boto:Token: None 2023-09-29T22:22:46.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4= 2023-09-29T22:22:46.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4=', 'Content-Length': '0'} 2023-09-29T22:22:46.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091cf43d235a071ce-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.744 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.744 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.744 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.744 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.745 DEBUG:boto:Method: GET 2023-09-29T22:22:46.745 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.745 DEBUG:boto:Data: 2023-09-29T22:22:46.745 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.745 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.745 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.745 DEBUG:boto:Params: {} 2023-09-29T22:22:46.745 DEBUG:boto:Token: None 2023-09-29T22:22:46.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:46.746 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4= 2023-09-29T22:22:46.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lT5CY3w+QMZrc41cl5ur3Bc4Qv4=', 'Content-Length': '0'} 2023-09-29T22:22:46.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af53bbc1074ac9d1-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.790 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.791 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.791 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.791 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.791 DEBUG:boto:Method: GET 2023-09-29T22:22:46.791 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.791 DEBUG:boto:Data: 2023-09-29T22:22:46.791 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.791 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.791 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.791 DEBUG:boto:Params: {} 2023-09-29T22:22:46.792 DEBUG:boto:Token: None 2023-09-29T22:22:46.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1972fa1b3162c88-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.794 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.794 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.795 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.795 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.795 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.795 DEBUG:boto:Method: GET 2023-09-29T22:22:46.795 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.795 DEBUG:boto:Data: 2023-09-29T22:22:46.795 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.795 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.795 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.796 DEBUG:boto:Params: {} 2023-09-29T22:22:46.796 DEBUG:boto:Token: None 2023-09-29T22:22:46.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.796 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492b7176619fc167-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.838 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.839 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.839 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.839 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.839 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.839 DEBUG:boto:Data: 2023-09-29T22:22:46.839 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.839 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.839 DEBUG:boto:Params: {} 2023-09-29T22:22:46.840 DEBUG:boto:Token: None 2023-09-29T22:22:46.840 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c80e14a308a4bdcf-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.843 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.843 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.843 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.843 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.843 DEBUG:boto:Data: 2023-09-29T22:22:46.843 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.843 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.843 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.844 DEBUG:boto:Params: {} 2023-09-29T22:22:46.844 DEBUG:boto:Token: None 2023-09-29T22:22:46.844 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.844 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084b8d14d70897c95-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.847 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.847 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.847 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.847 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.847 DEBUG:boto:Method: GET 2023-09-29T22:22:46.847 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.847 DEBUG:boto:Data: 2023-09-29T22:22:46.848 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.848 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.848 DEBUG:boto:Params: {} 2023-09-29T22:22:46.848 DEBUG:boto:Token: None 2023-09-29T22:22:46.848 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.848 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb04a2948e8c5d28-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.850 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.850 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.850 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.850 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.851 DEBUG:boto:Method: GET 2023-09-29T22:22:46.851 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.851 DEBUG:boto:Data: 2023-09-29T22:22:46.851 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.851 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.851 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.851 DEBUG:boto:Params: {} 2023-09-29T22:22:46.851 DEBUG:boto:Token: None 2023-09-29T22:22:46.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ccc1af144cbcaaf-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.854 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.854 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.854 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.854 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.854 DEBUG:boto:Method: GET 2023-09-29T22:22:46.854 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.854 DEBUG:boto:Data: 2023-09-29T22:22:46.854 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.855 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.855 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.855 DEBUG:boto:Params: {} 2023-09-29T22:22:46.855 DEBUG:boto:Token: None 2023-09-29T22:22:46.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.855 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf95b869fea85e58-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.857 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.858 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.858 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.858 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.858 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.858 DEBUG:boto:Method: GET 2023-09-29T22:22:46.858 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.859 DEBUG:boto:Data: 2023-09-29T22:22:46.859 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.859 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.859 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.859 DEBUG:boto:Params: {} 2023-09-29T22:22:46.859 DEBUG:boto:Token: None 2023-09-29T22:22:46.859 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.859 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed392d4fd4bace97-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.862 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.862 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.863 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.863 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.863 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.863 DEBUG:boto:Method: GET 2023-09-29T22:22:46.863 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.863 DEBUG:boto:Data: 2023-09-29T22:22:46.863 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.863 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.863 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.863 DEBUG:boto:Params: {} 2023-09-29T22:22:46.864 DEBUG:boto:Token: None 2023-09-29T22:22:46.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q= 2023-09-29T22:22:46.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q=', 'Content-Length': '0'} 2023-09-29T22:22:46.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad1ebac344768421-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.866 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.866 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.866 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.866 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.866 DEBUG:boto:Method: GET 2023-09-29T22:22:46.866 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.866 DEBUG:boto:Data: 2023-09-29T22:22:46.866 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.867 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.867 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.867 DEBUG:boto:Params: {} 2023-09-29T22:22:46.867 DEBUG:boto:Token: None 2023-09-29T22:22:46.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q= 2023-09-29T22:22:46.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q=', 'Content-Length': '0'} 2023-09-29T22:22:46.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000281e9f0c443dc9ea-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.869 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.869 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.869 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.870 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.870 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.870 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.870 DEBUG:boto:Data: 2023-09-29T22:22:46.870 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.870 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.870 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.870 DEBUG:boto:Params: {} 2023-09-29T22:22:46.870 DEBUG:boto:Token: None 2023-09-29T22:22:46.870 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.871 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2NP1Jexa6AGclERvYC53FUC7dBI= 2023-09-29T22:22:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2NP1Jexa6AGclERvYC53FUC7dBI=', 'Content-Length': '0'} 2023-09-29T22:22:46.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031d92f548c00a25a-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.873 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.873 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.873 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.873 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.873 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.873 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.873 DEBUG:boto:Data: 2023-09-29T22:22:46.873 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.873 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.873 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.874 DEBUG:boto:Params: {} 2023-09-29T22:22:46.874 DEBUG:boto:Token: None 2023-09-29T22:22:46.874 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.874 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2NP1Jexa6AGclERvYC53FUC7dBI= 2023-09-29T22:22:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2NP1Jexa6AGclERvYC53FUC7dBI=', 'Content-Length': '0'} 2023-09-29T22:22:46.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c5aff1102d535b6-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.876 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.876 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.876 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.876 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.876 DEBUG:boto:Method: GET 2023-09-29T22:22:46.876 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.876 DEBUG:boto:Data: 2023-09-29T22:22:46.876 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.877 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.877 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.877 DEBUG:boto:Params: {} 2023-09-29T22:22:46.877 DEBUG:boto:Token: None 2023-09-29T22:22:46.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.877 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q= 2023-09-29T22:22:46.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q=', 'Content-Length': '0'} 2023-09-29T22:22:46.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ed506ea2c1ec308-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.879 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.879 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.879 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.879 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.880 DEBUG:boto:Method: GET 2023-09-29T22:22:46.880 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.880 DEBUG:boto:Data: 2023-09-29T22:22:46.880 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.880 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.880 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.880 DEBUG:boto:Params: {} 2023-09-29T22:22:46.880 DEBUG:boto:Token: None 2023-09-29T22:22:46.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:46.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q= 2023-09-29T22:22:46.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MtSgjMBRFpUti3TWw3QnvDamQ3Q=', 'Content-Length': '0'} 2023-09-29T22:22:46.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003641eca86b295d28-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.883 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.883 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.883 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.883 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.883 DEBUG:boto:Method: GET 2023-09-29T22:22:46.883 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.883 DEBUG:boto:Data: 2023-09-29T22:22:46.884 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.884 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.884 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.884 DEBUG:boto:Params: {} 2023-09-29T22:22:46.884 DEBUG:boto:Token: None 2023-09-29T22:22:46.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7268d33cb698826-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.930 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.931 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.931 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.931 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.931 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.931 DEBUG:boto:Method: GET 2023-09-29T22:22:46.931 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.931 DEBUG:boto:Data: 2023-09-29T22:22:46.932 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.932 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.932 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.932 DEBUG:boto:Params: {} 2023-09-29T22:22:46.932 DEBUG:boto:Token: None 2023-09-29T22:22:46.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeeca0b526a6da87-0065174e36-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.935 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.935 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.935 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.936 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.936 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.936 DEBUG:boto:Data: 2023-09-29T22:22:46.936 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.936 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.936 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.936 DEBUG:boto:Params: {} 2023-09-29T22:22:46.936 DEBUG:boto:Token: None 2023-09-29T22:22:46.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085ea68872941a0ed-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.941 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.941 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.941 DEBUG:boto:Method: HEAD 2023-09-29T22:22:46.941 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:46.941 DEBUG:boto:Data: 2023-09-29T22:22:46.941 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.942 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.942 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.942 DEBUG:boto:Params: {} 2023-09-29T22:22:46.942 DEBUG:boto:Token: None 2023-09-29T22:22:46.942 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj 2023-09-29T22:22:46.942 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU= 2023-09-29T22:22:46.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:69pMxjzqefjDjEa1oiIiA9ps/aU=', 'Content-Length': '0'} 2023-09-29T22:22:46.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3f7e2dd3edc8ef-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:46.945 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.945 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.945 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.945 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.945 DEBUG:boto:Method: GET 2023-09-29T22:22:46.946 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.946 DEBUG:boto:Data: 2023-09-29T22:22:46.946 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.946 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.946 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.946 DEBUG:boto:Params: {} 2023-09-29T22:22:46.946 DEBUG:boto:Token: None 2023-09-29T22:22:46.946 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.947 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ebff8783fa4f487-0065174e36-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.948 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:46.948 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:46.949 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.949 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.949 DEBUG:boto:Method: GET 2023-09-29T22:22:46.949 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.949 DEBUG:boto:Data: 2023-09-29T22:22:46.949 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.949 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.949 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.949 DEBUG:boto:Params: {} 2023-09-29T22:22:46.950 DEBUG:boto:Token: None 2023-09-29T22:22:46.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:46.950 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk= 2023-09-29T22:22:46.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VR5A10apUYuW8hrgR5yIm3y8xLk=', 'Content-Length': '0'} 2023-09-29T22:22:46.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018eb7f7ab64f25c4-0065174e36-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.994 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.994 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.995 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.995 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.995 DEBUG:boto:Method: GET 2023-09-29T22:22:46.995 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.995 DEBUG:boto:Data: 2023-09-29T22:22:46.995 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:46.995 DEBUG:boto:Port: 8001 2023-09-29T22:22:46.995 DEBUG:boto:Params: {} 2023-09-29T22:22:46.995 DEBUG:boto:Token: None 2023-09-29T22:22:46.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:46.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:46.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:46.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e3ba167e0ea99f-0065174e36-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:46.998 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:46.998 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:46.998 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:46.998 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.999 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.999 DEBUG:boto:Method: GET 2023-09-29T22:22:46.999 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:46.999 DEBUG:boto:Data: 2023-09-29T22:22:46.999 DEBUG:boto:Headers: {} 2023-09-29T22:22:46.999 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:46.999 DEBUG:boto:Port: 8000 2023-09-29T22:22:46.999 DEBUG:boto:Params: {} 2023-09-29T22:22:46.999 DEBUG:boto:Token: None 2023-09-29T22:22:46.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:46 GMT /dmuhsu-19/ 2023-09-29T22:22:47.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o= 2023-09-29T22:22:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:46 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lDulYY+D8CenfB1zx3ou0cQbX/o=', 'Content-Length': '0'} 2023-09-29T22:22:47.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b3faf1b374302b-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.002 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.002 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.003 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.003 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.003 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.003 DEBUG:boto:Method: GET 2023-09-29T22:22:47.003 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.003 DEBUG:boto:Data: 2023-09-29T22:22:47.003 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.003 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.003 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.003 DEBUG:boto:Params: {} 2023-09-29T22:22:47.004 DEBUG:boto:Token: None 2023-09-29T22:22:47.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8= 2023-09-29T22:22:47.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8=', 'Content-Length': '0'} 2023-09-29T22:22:47.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008454e0942f76fa0f-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.005 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.006 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.006 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.006 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.006 DEBUG:boto:Method: GET 2023-09-29T22:22:47.006 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.006 DEBUG:boto:Data: 2023-09-29T22:22:47.006 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.006 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.006 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.007 DEBUG:boto:Params: {} 2023-09-29T22:22:47.007 DEBUG:boto:Token: None 2023-09-29T22:22:47.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8= 2023-09-29T22:22:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8=', 'Content-Length': '0'} 2023-09-29T22:22:47.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbd730cc8a302230-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.009 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.009 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.009 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.009 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.009 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.009 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.009 DEBUG:boto:Data: 2023-09-29T22:22:47.009 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.009 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.010 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.010 DEBUG:boto:Params: {} 2023-09-29T22:22:47.010 DEBUG:boto:Token: None 2023-09-29T22:22:47.010 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.010 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pj9Mir/Labl3Ll/Rx9BcYOdmobM= 2023-09-29T22:22:47.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pj9Mir/Labl3Ll/Rx9BcYOdmobM=', 'Content-Length': '0'} 2023-09-29T22:22:47.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029f8e8dc011c91f1-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.012 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.012 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.012 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.012 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.012 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.012 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.012 DEBUG:boto:Data: 2023-09-29T22:22:47.012 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.012 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.013 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.013 DEBUG:boto:Params: {} 2023-09-29T22:22:47.013 DEBUG:boto:Token: None 2023-09-29T22:22:47.013 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.013 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pj9Mir/Labl3Ll/Rx9BcYOdmobM= 2023-09-29T22:22:47.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pj9Mir/Labl3Ll/Rx9BcYOdmobM=', 'Content-Length': '0'} 2023-09-29T22:22:47.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c2d4943f196b0dc-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.015 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.015 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.015 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.015 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.015 DEBUG:boto:Method: GET 2023-09-29T22:22:47.016 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.016 DEBUG:boto:Data: 2023-09-29T22:22:47.016 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.016 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.016 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.016 DEBUG:boto:Params: {} 2023-09-29T22:22:47.016 DEBUG:boto:Token: None 2023-09-29T22:22:47.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8= 2023-09-29T22:22:47.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8=', 'Content-Length': '0'} 2023-09-29T22:22:47.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e30aa511138f24ef-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.018 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.018 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.018 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.018 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.018 DEBUG:boto:Method: GET 2023-09-29T22:22:47.019 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.019 DEBUG:boto:Data: 2023-09-29T22:22:47.019 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.019 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.019 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.019 DEBUG:boto:Params: {} 2023-09-29T22:22:47.019 DEBUG:boto:Token: None 2023-09-29T22:22:47.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:47.019 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8= 2023-09-29T22:22:47.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uit6EF69yAGKdPpZWgl40slsjA8=', 'Content-Length': '0'} 2023-09-29T22:22:47.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1faae6859b8f985-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.066 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.066 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.067 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.067 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.067 DEBUG:boto:Method: GET 2023-09-29T22:22:47.067 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.067 DEBUG:boto:Data: 2023-09-29T22:22:47.067 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.067 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.067 DEBUG:boto:Params: {} 2023-09-29T22:22:47.067 DEBUG:boto:Token: None 2023-09-29T22:22:47.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad8b77b088a4dec-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.070 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.070 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.071 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.071 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.071 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.071 DEBUG:boto:Method: GET 2023-09-29T22:22:47.071 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.071 DEBUG:boto:Data: 2023-09-29T22:22:47.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.071 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.072 DEBUG:boto:Params: {} 2023-09-29T22:22:47.072 DEBUG:boto:Token: None 2023-09-29T22:22:47.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94b6269a3297be5-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.114 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.114 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.115 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.115 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.115 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.115 DEBUG:boto:Data: 2023-09-29T22:22:47.115 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.115 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.115 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.115 DEBUG:boto:Params: {} 2023-09-29T22:22:47.116 DEBUG:boto:Token: None 2023-09-29T22:22:47.116 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a236a532da574bac-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.119 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.119 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.119 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.119 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.119 DEBUG:boto:Data: 2023-09-29T22:22:47.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.119 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.119 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.119 DEBUG:boto:Params: {} 2023-09-29T22:22:47.120 DEBUG:boto:Token: None 2023-09-29T22:22:47.120 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de927915ca50b726-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.123 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.123 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.123 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.123 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.123 DEBUG:boto:Method: GET 2023-09-29T22:22:47.123 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.123 DEBUG:boto:Data: 2023-09-29T22:22:47.123 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.123 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.124 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.124 DEBUG:boto:Params: {} 2023-09-29T22:22:47.124 DEBUG:boto:Token: None 2023-09-29T22:22:47.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000877d842564e9b9dc-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.166 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.166 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.167 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.167 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.167 DEBUG:boto:Method: GET 2023-09-29T22:22:47.167 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.167 DEBUG:boto:Data: 2023-09-29T22:22:47.167 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.167 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.167 DEBUG:boto:Params: {} 2023-09-29T22:22:47.167 DEBUG:boto:Token: None 2023-09-29T22:22:47.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1900556066056c6-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.170 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.170 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.170 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.170 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.170 DEBUG:boto:Method: GET 2023-09-29T22:22:47.170 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.170 DEBUG:boto:Data: 2023-09-29T22:22:47.170 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.170 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.171 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.171 DEBUG:boto:Params: {} 2023-09-29T22:22:47.171 DEBUG:boto:Token: None 2023-09-29T22:22:47.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.171 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6840ed655e0391-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.174 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.174 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.174 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.174 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.174 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.174 DEBUG:boto:Method: GET 2023-09-29T22:22:47.175 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.175 DEBUG:boto:Data: 2023-09-29T22:22:47.175 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.175 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.175 DEBUG:boto:Params: {} 2023-09-29T22:22:47.175 DEBUG:boto:Token: None 2023-09-29T22:22:47.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45d089a7e4ef67a-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.177 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.178 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.178 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.178 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.178 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.178 DEBUG:boto:Method: GET 2023-09-29T22:22:47.179 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.179 DEBUG:boto:Data: 2023-09-29T22:22:47.179 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.179 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.179 DEBUG:boto:Params: {} 2023-09-29T22:22:47.179 DEBUG:boto:Token: None 2023-09-29T22:22:47.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI= 2023-09-29T22:22:47.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI=', 'Content-Length': '0'} 2023-09-29T22:22:47.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000691a38b74c9bbdf2-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.181 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.181 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.181 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.181 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.182 DEBUG:boto:Method: GET 2023-09-29T22:22:47.182 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.182 DEBUG:boto:Data: 2023-09-29T22:22:47.182 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.182 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.182 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.182 DEBUG:boto:Params: {} 2023-09-29T22:22:47.182 DEBUG:boto:Token: None 2023-09-29T22:22:47.182 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI= 2023-09-29T22:22:47.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI=', 'Content-Length': '0'} 2023-09-29T22:22:47.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f74b968129e195bb-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.184 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.184 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.184 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.184 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.185 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.185 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.185 DEBUG:boto:Data: 2023-09-29T22:22:47.185 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.185 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.185 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.185 DEBUG:boto:Params: {} 2023-09-29T22:22:47.185 DEBUG:boto:Token: None 2023-09-29T22:22:47.185 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.186 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tGTjNwo5CagBx9OVAca57zR/EyY= 2023-09-29T22:22:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tGTjNwo5CagBx9OVAca57zR/EyY=', 'Content-Length': '0'} 2023-09-29T22:22:47.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000398073fe5a7d8158-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.187 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.187 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.187 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.187 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.188 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.188 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.188 DEBUG:boto:Data: 2023-09-29T22:22:47.188 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.188 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.188 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.188 DEBUG:boto:Params: {} 2023-09-29T22:22:47.188 DEBUG:boto:Token: None 2023-09-29T22:22:47.189 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.189 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tGTjNwo5CagBx9OVAca57zR/EyY= 2023-09-29T22:22:47.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tGTjNwo5CagBx9OVAca57zR/EyY=', 'Content-Length': '0'} 2023-09-29T22:22:47.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000267909f80f177472-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.191 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.191 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.191 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.191 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.191 DEBUG:boto:Method: GET 2023-09-29T22:22:47.191 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.191 DEBUG:boto:Data: 2023-09-29T22:22:47.191 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.191 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.191 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.192 DEBUG:boto:Params: {} 2023-09-29T22:22:47.192 DEBUG:boto:Token: None 2023-09-29T22:22:47.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.192 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI= 2023-09-29T22:22:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI=', 'Content-Length': '0'} 2023-09-29T22:22:47.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022b7dd0a1f42a9cc-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.194 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.194 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.194 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.194 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.195 DEBUG:boto:Method: GET 2023-09-29T22:22:47.195 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.195 DEBUG:boto:Data: 2023-09-29T22:22:47.195 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.195 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.195 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.195 DEBUG:boto:Params: {} 2023-09-29T22:22:47.195 DEBUG:boto:Token: None 2023-09-29T22:22:47.195 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:47.195 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI= 2023-09-29T22:22:47.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fZoKDg8CHfAE9ax7DC2kLLq1+vI=', 'Content-Length': '0'} 2023-09-29T22:22:47.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1e7fda0a3dc3c29-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.197 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.197 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.197 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.198 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.198 DEBUG:boto:Method: GET 2023-09-29T22:22:47.198 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.198 DEBUG:boto:Data: 2023-09-29T22:22:47.198 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.198 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.198 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.198 DEBUG:boto:Params: {} 2023-09-29T22:22:47.198 DEBUG:boto:Token: None 2023-09-29T22:22:47.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.199 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786a9ae5dc163c3e-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.201 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.201 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.201 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.201 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.201 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.202 DEBUG:boto:Method: GET 2023-09-29T22:22:47.202 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.202 DEBUG:boto:Data: 2023-09-29T22:22:47.202 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.202 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.202 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.202 DEBUG:boto:Params: {} 2023-09-29T22:22:47.202 DEBUG:boto:Token: None 2023-09-29T22:22:47.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.202 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db046e7093ceef29-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.205 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.205 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.205 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.205 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.205 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.206 DEBUG:boto:Data: 2023-09-29T22:22:47.206 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.206 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.206 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.206 DEBUG:boto:Params: {} 2023-09-29T22:22:47.206 DEBUG:boto:Token: None 2023-09-29T22:22:47.206 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.206 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009df7622540bdfcea-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.209 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.209 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.209 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.209 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.210 DEBUG:boto:Data: 2023-09-29T22:22:47.210 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.210 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.210 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.210 DEBUG:boto:Params: {} 2023-09-29T22:22:47.210 DEBUG:boto:Token: None 2023-09-29T22:22:47.210 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.210 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079e2a200b46b1c8e-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.213 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.213 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.213 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.213 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.213 DEBUG:boto:Method: GET 2023-09-29T22:22:47.214 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.214 DEBUG:boto:Data: 2023-09-29T22:22:47.214 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.214 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.214 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.214 DEBUG:boto:Params: {} 2023-09-29T22:22:47.214 DEBUG:boto:Token: None 2023-09-29T22:22:47.214 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.214 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b400b63be578272b-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.216 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.216 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.216 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.216 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.216 DEBUG:boto:Method: GET 2023-09-29T22:22:47.217 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.217 DEBUG:boto:Data: 2023-09-29T22:22:47.217 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.217 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.217 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.217 DEBUG:boto:Params: {} 2023-09-29T22:22:47.217 DEBUG:boto:Token: None 2023-09-29T22:22:47.217 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.217 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d50dd261a8b62d2-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.219 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.219 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.219 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.219 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.219 DEBUG:boto:Method: GET 2023-09-29T22:22:47.219 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.220 DEBUG:boto:Data: 2023-09-29T22:22:47.220 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.220 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.220 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.220 DEBUG:boto:Params: {} 2023-09-29T22:22:47.220 DEBUG:boto:Token: None 2023-09-29T22:22:47.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba2f4de206a6ae1-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.223 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.223 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.223 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.224 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.224 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.224 DEBUG:boto:Method: GET 2023-09-29T22:22:47.224 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.224 DEBUG:boto:Data: 2023-09-29T22:22:47.224 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.224 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.224 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.224 DEBUG:boto:Params: {} 2023-09-29T22:22:47.224 DEBUG:boto:Token: None 2023-09-29T22:22:47.225 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.225 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac3248fd8210e62-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.270 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.271 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.271 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.271 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.271 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.271 DEBUG:boto:Method: GET 2023-09-29T22:22:47.271 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.272 DEBUG:boto:Data: 2023-09-29T22:22:47.272 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.272 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.272 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.272 DEBUG:boto:Params: {} 2023-09-29T22:22:47.272 DEBUG:boto:Token: None 2023-09-29T22:22:47.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0= 2023-09-29T22:22:47.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0=', 'Content-Length': '0'} 2023-09-29T22:22:47.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076bf75d9cca86110-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.274 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.275 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.275 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.275 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.275 DEBUG:boto:Method: GET 2023-09-29T22:22:47.275 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.275 DEBUG:boto:Data: 2023-09-29T22:22:47.275 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.275 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.275 DEBUG:boto:Params: {} 2023-09-29T22:22:47.276 DEBUG:boto:Token: None 2023-09-29T22:22:47.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0= 2023-09-29T22:22:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0=', 'Content-Length': '0'} 2023-09-29T22:22:47.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a696731ea4689a61-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.277 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.278 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.278 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.278 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.278 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.278 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.278 DEBUG:boto:Data: 2023-09-29T22:22:47.278 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.278 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.278 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.279 DEBUG:boto:Params: {} 2023-09-29T22:22:47.279 DEBUG:boto:Token: None 2023-09-29T22:22:47.279 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.279 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRXLGoWbnmSTiw9r2Jtjtt1D0kM= 2023-09-29T22:22:47.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRXLGoWbnmSTiw9r2Jtjtt1D0kM=', 'Content-Length': '0'} 2023-09-29T22:22:47.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddc31455edcf56d1-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.281 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.281 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.281 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.281 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.281 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.281 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.281 DEBUG:boto:Data: 2023-09-29T22:22:47.282 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.282 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.282 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.282 DEBUG:boto:Params: {} 2023-09-29T22:22:47.282 DEBUG:boto:Token: None 2023-09-29T22:22:47.282 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.282 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRXLGoWbnmSTiw9r2Jtjtt1D0kM= 2023-09-29T22:22:47.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRXLGoWbnmSTiw9r2Jtjtt1D0kM=', 'Content-Length': '0'} 2023-09-29T22:22:47.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b15438d55b70714c-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.284 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.284 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.284 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.284 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.285 DEBUG:boto:Method: GET 2023-09-29T22:22:47.285 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.285 DEBUG:boto:Data: 2023-09-29T22:22:47.285 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.285 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.285 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.285 DEBUG:boto:Params: {} 2023-09-29T22:22:47.285 DEBUG:boto:Token: None 2023-09-29T22:22:47.285 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.286 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0= 2023-09-29T22:22:47.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0=', 'Content-Length': '0'} 2023-09-29T22:22:47.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d24399af5f56488-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.287 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.287 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.287 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.288 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.288 DEBUG:boto:Method: GET 2023-09-29T22:22:47.288 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.288 DEBUG:boto:Data: 2023-09-29T22:22:47.288 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.288 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.288 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.288 DEBUG:boto:Params: {} 2023-09-29T22:22:47.288 DEBUG:boto:Token: None 2023-09-29T22:22:47.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:47.289 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0= 2023-09-29T22:22:47.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lv+p90344TvNj54Kwyg8nxoroj0=', 'Content-Length': '0'} 2023-09-29T22:22:47.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2a266eb9d9ffd2d-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.334 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.335 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.335 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.335 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.335 DEBUG:boto:Method: GET 2023-09-29T22:22:47.335 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.335 DEBUG:boto:Data: 2023-09-29T22:22:47.335 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.335 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.335 DEBUG:boto:Params: {} 2023-09-29T22:22:47.336 DEBUG:boto:Token: None 2023-09-29T22:22:47.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c4d619ce76911d-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.339 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.339 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.339 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.339 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.339 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.340 DEBUG:boto:Method: GET 2023-09-29T22:22:47.340 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.340 DEBUG:boto:Data: 2023-09-29T22:22:47.340 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.340 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.340 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.340 DEBUG:boto:Params: {} 2023-09-29T22:22:47.340 DEBUG:boto:Token: None 2023-09-29T22:22:47.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.341 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002170fec87d0ae50e-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.343 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.343 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.343 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.343 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.343 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.344 DEBUG:boto:Data: 2023-09-29T22:22:47.344 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.344 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.344 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.344 DEBUG:boto:Params: {} 2023-09-29T22:22:47.344 DEBUG:boto:Token: None 2023-09-29T22:22:47.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a6f4e38f93413d4-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.347 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.347 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.347 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.347 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.347 DEBUG:boto:Data: 2023-09-29T22:22:47.348 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.348 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.348 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.348 DEBUG:boto:Params: {} 2023-09-29T22:22:47.348 DEBUG:boto:Token: None 2023-09-29T22:22:47.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.348 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017eb5c1e5a2c9680-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.353 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.353 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.353 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.353 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.354 DEBUG:boto:Method: GET 2023-09-29T22:22:47.354 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.354 DEBUG:boto:Data: 2023-09-29T22:22:47.354 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.354 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.354 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.354 DEBUG:boto:Params: {} 2023-09-29T22:22:47.354 DEBUG:boto:Token: None 2023-09-29T22:22:47.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.355 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd9bfd1b219f0c21-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.398 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.398 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.398 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.399 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.399 DEBUG:boto:Method: GET 2023-09-29T22:22:47.399 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.399 DEBUG:boto:Data: 2023-09-29T22:22:47.399 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.399 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.399 DEBUG:boto:Params: {} 2023-09-29T22:22:47.399 DEBUG:boto:Token: None 2023-09-29T22:22:47.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e13bda0b3be46158-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.401 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.402 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.402 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.402 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.402 DEBUG:boto:Method: GET 2023-09-29T22:22:47.402 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.402 DEBUG:boto:Data: 2023-09-29T22:22:47.402 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.402 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.402 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.403 DEBUG:boto:Params: {} 2023-09-29T22:22:47.403 DEBUG:boto:Token: None 2023-09-29T22:22:47.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.403 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b7c31abbf69346-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.405 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.406 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.406 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.406 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.406 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.406 DEBUG:boto:Method: GET 2023-09-29T22:22:47.406 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.406 DEBUG:boto:Data: 2023-09-29T22:22:47.406 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.406 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.407 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.407 DEBUG:boto:Params: {} 2023-09-29T22:22:47.407 DEBUG:boto:Token: None 2023-09-29T22:22:47.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.407 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb5e68c7790e4f6-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.410 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.410 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.410 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.410 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.411 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.411 DEBUG:boto:Method: GET 2023-09-29T22:22:47.411 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.411 DEBUG:boto:Data: 2023-09-29T22:22:47.411 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.411 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.411 DEBUG:boto:Params: {} 2023-09-29T22:22:47.411 DEBUG:boto:Token: None 2023-09-29T22:22:47.411 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4= 2023-09-29T22:22:47.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4=', 'Content-Length': '0'} 2023-09-29T22:22:47.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000685bc35c9f808186-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.414 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.414 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.414 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.414 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.414 DEBUG:boto:Method: GET 2023-09-29T22:22:47.414 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.414 DEBUG:boto:Data: 2023-09-29T22:22:47.414 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.414 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.415 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.415 DEBUG:boto:Params: {} 2023-09-29T22:22:47.415 DEBUG:boto:Token: None 2023-09-29T22:22:47.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4= 2023-09-29T22:22:47.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4=', 'Content-Length': '0'} 2023-09-29T22:22:47.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008b7b42a864c1e60-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.458 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.459 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.459 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.459 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.459 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.459 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.459 DEBUG:boto:Data: 2023-09-29T22:22:47.459 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.459 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.460 DEBUG:boto:Params: {} 2023-09-29T22:22:47.460 DEBUG:boto:Token: None 2023-09-29T22:22:47.460 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RYJPDmKHeOXw82mnpeIbjH7psDw= 2023-09-29T22:22:47.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RYJPDmKHeOXw82mnpeIbjH7psDw=', 'Content-Length': '0'} 2023-09-29T22:22:47.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9e0c1855e02a0a7-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.462 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.462 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.462 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.462 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.462 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.462 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.462 DEBUG:boto:Data: 2023-09-29T22:22:47.463 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.463 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.463 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.463 DEBUG:boto:Params: {} 2023-09-29T22:22:47.463 DEBUG:boto:Token: None 2023-09-29T22:22:47.463 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.463 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RYJPDmKHeOXw82mnpeIbjH7psDw= 2023-09-29T22:22:47.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RYJPDmKHeOXw82mnpeIbjH7psDw=', 'Content-Length': '0'} 2023-09-29T22:22:47.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064528a2bbc91e238-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.465 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.465 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.465 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.465 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.465 DEBUG:boto:Method: GET 2023-09-29T22:22:47.465 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.466 DEBUG:boto:Data: 2023-09-29T22:22:47.466 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.466 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.466 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.466 DEBUG:boto:Params: {} 2023-09-29T22:22:47.466 DEBUG:boto:Token: None 2023-09-29T22:22:47.466 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.466 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4= 2023-09-29T22:22:47.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4=', 'Content-Length': '0'} 2023-09-29T22:22:47.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac748a55fc7cbcf1-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.468 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.468 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.468 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.468 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.468 DEBUG:boto:Method: GET 2023-09-29T22:22:47.468 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.469 DEBUG:boto:Data: 2023-09-29T22:22:47.469 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.469 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.469 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.469 DEBUG:boto:Params: {} 2023-09-29T22:22:47.469 DEBUG:boto:Token: None 2023-09-29T22:22:47.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:47.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4= 2023-09-29T22:22:47.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PZ6mhORneiFVAEMSXfBJtHsKxd4=', 'Content-Length': '0'} 2023-09-29T22:22:47.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000001dffcae290d1cd-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.514 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.514 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.515 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.515 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.515 DEBUG:boto:Method: GET 2023-09-29T22:22:47.515 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.515 DEBUG:boto:Data: 2023-09-29T22:22:47.515 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.515 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.515 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.515 DEBUG:boto:Params: {} 2023-09-29T22:22:47.515 DEBUG:boto:Token: None 2023-09-29T22:22:47.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a501a92a0c0e1a95-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.518 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.518 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.518 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.519 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.519 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.519 DEBUG:boto:Method: GET 2023-09-29T22:22:47.519 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.519 DEBUG:boto:Data: 2023-09-29T22:22:47.519 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.519 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.519 DEBUG:boto:Params: {} 2023-09-29T22:22:47.519 DEBUG:boto:Token: None 2023-09-29T22:22:47.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.520 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac631aab0b4df471-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.522 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.523 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.523 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.523 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.523 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.523 DEBUG:boto:Data: 2023-09-29T22:22:47.523 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.523 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.523 DEBUG:boto:Params: {} 2023-09-29T22:22:47.524 DEBUG:boto:Token: None 2023-09-29T22:22:47.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009659e933f8556acb-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.526 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.526 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.527 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.527 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.527 DEBUG:boto:Data: 2023-09-29T22:22:47.527 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.527 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.527 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.527 DEBUG:boto:Params: {} 2023-09-29T22:22:47.527 DEBUG:boto:Token: None 2023-09-29T22:22:47.527 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.528 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cab68e2e03ad673-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.530 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.530 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.531 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.531 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.531 DEBUG:boto:Method: GET 2023-09-29T22:22:47.531 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.531 DEBUG:boto:Data: 2023-09-29T22:22:47.531 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.531 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.531 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.531 DEBUG:boto:Params: {} 2023-09-29T22:22:47.531 DEBUG:boto:Token: None 2023-09-29T22:22:47.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a92548b0f8fc50da-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.534 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.534 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.534 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.534 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.534 DEBUG:boto:Method: GET 2023-09-29T22:22:47.534 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.534 DEBUG:boto:Data: 2023-09-29T22:22:47.535 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.535 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.535 DEBUG:boto:Params: {} 2023-09-29T22:22:47.535 DEBUG:boto:Token: None 2023-09-29T22:22:47.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.535 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2666c071ae626ea-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.578 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.578 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.578 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.579 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.579 DEBUG:boto:Method: GET 2023-09-29T22:22:47.579 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.579 DEBUG:boto:Data: 2023-09-29T22:22:47.579 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.579 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.579 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.579 DEBUG:boto:Params: {} 2023-09-29T22:22:47.579 DEBUG:boto:Token: None 2023-09-29T22:22:47.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.580 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd12bf733f67479c-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.582 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.582 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.583 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.583 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.583 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.583 DEBUG:boto:Method: GET 2023-09-29T22:22:47.583 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.583 DEBUG:boto:Data: 2023-09-29T22:22:47.583 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.583 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.583 DEBUG:boto:Params: {} 2023-09-29T22:22:47.584 DEBUG:boto:Token: None 2023-09-29T22:22:47.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd0c45ec0d575d08-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.586 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.587 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.587 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.587 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.587 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.587 DEBUG:boto:Method: GET 2023-09-29T22:22:47.587 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.587 DEBUG:boto:Data: 2023-09-29T22:22:47.587 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.587 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.588 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.588 DEBUG:boto:Params: {} 2023-09-29T22:22:47.588 DEBUG:boto:Token: None 2023-09-29T22:22:47.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw= 2023-09-29T22:22:47.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw=', 'Content-Length': '0'} 2023-09-29T22:22:47.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c03a07db42ea5df-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.590 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.590 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.590 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.590 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.590 DEBUG:boto:Method: GET 2023-09-29T22:22:47.591 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.591 DEBUG:boto:Data: 2023-09-29T22:22:47.591 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.591 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.591 DEBUG:boto:Params: {} 2023-09-29T22:22:47.591 DEBUG:boto:Token: None 2023-09-29T22:22:47.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw= 2023-09-29T22:22:47.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw=', 'Content-Length': '0'} 2023-09-29T22:22:47.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fc42de11232171b-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.593 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.594 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.594 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.594 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.594 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.594 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.594 DEBUG:boto:Data: 2023-09-29T22:22:47.594 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.594 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.594 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.595 DEBUG:boto:Params: {} 2023-09-29T22:22:47.595 DEBUG:boto:Token: None 2023-09-29T22:22:47.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.595 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7UjkDqlB8rMN0f7IHBDgaM8JeSg= 2023-09-29T22:22:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7UjkDqlB8rMN0f7IHBDgaM8JeSg=', 'Content-Length': '0'} 2023-09-29T22:22:47.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a71c90a9ec4b87be-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.597 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.597 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.597 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.597 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.597 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.597 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.598 DEBUG:boto:Data: 2023-09-29T22:22:47.598 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.598 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.598 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.598 DEBUG:boto:Params: {} 2023-09-29T22:22:47.598 DEBUG:boto:Token: None 2023-09-29T22:22:47.598 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7UjkDqlB8rMN0f7IHBDgaM8JeSg= 2023-09-29T22:22:47.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7UjkDqlB8rMN0f7IHBDgaM8JeSg=', 'Content-Length': '0'} 2023-09-29T22:22:47.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000025a0b5c41d7a0a7-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.600 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.600 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.601 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.601 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.601 DEBUG:boto:Method: GET 2023-09-29T22:22:47.601 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.601 DEBUG:boto:Data: 2023-09-29T22:22:47.601 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.601 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.601 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.601 DEBUG:boto:Params: {} 2023-09-29T22:22:47.601 DEBUG:boto:Token: None 2023-09-29T22:22:47.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw= 2023-09-29T22:22:47.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw=', 'Content-Length': '0'} 2023-09-29T22:22:47.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f0c6c5cb52dd98-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.604 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.604 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.604 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.604 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.604 DEBUG:boto:Method: GET 2023-09-29T22:22:47.604 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.604 DEBUG:boto:Data: 2023-09-29T22:22:47.605 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.605 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.605 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.605 DEBUG:boto:Params: {} 2023-09-29T22:22:47.605 DEBUG:boto:Token: None 2023-09-29T22:22:47.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:47.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw= 2023-09-29T22:22:47.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OUSEr7QyvCu3KPC1gXGbrjYhDbw=', 'Content-Length': '0'} 2023-09-29T22:22:47.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001852988dcbf11da5-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.650 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.650 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.651 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.651 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.651 DEBUG:boto:Method: GET 2023-09-29T22:22:47.651 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.651 DEBUG:boto:Data: 2023-09-29T22:22:47.651 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.651 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.651 DEBUG:boto:Params: {} 2023-09-29T22:22:47.652 DEBUG:boto:Token: None 2023-09-29T22:22:47.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc22ad0cc0cd6c1-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.655 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.655 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.655 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.655 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.655 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.655 DEBUG:boto:Method: GET 2023-09-29T22:22:47.655 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.656 DEBUG:boto:Data: 2023-09-29T22:22:47.656 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.656 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.656 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.656 DEBUG:boto:Params: {} 2023-09-29T22:22:47.656 DEBUG:boto:Token: None 2023-09-29T22:22:47.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b52b8969a60b974-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.659 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.659 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.659 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.659 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.659 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.659 DEBUG:boto:Data: 2023-09-29T22:22:47.659 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.660 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.660 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.660 DEBUG:boto:Params: {} 2023-09-29T22:22:47.660 DEBUG:boto:Token: None 2023-09-29T22:22:47.660 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000048943db8d69437-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.663 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.663 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.663 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.663 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.663 DEBUG:boto:Data: 2023-09-29T22:22:47.663 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.663 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.664 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.664 DEBUG:boto:Params: {} 2023-09-29T22:22:47.664 DEBUG:boto:Token: None 2023-09-29T22:22:47.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005655e12be70df9df-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.667 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.667 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.667 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.667 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.667 DEBUG:boto:Method: GET 2023-09-29T22:22:47.667 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.667 DEBUG:boto:Data: 2023-09-29T22:22:47.667 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.668 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.668 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.668 DEBUG:boto:Params: {} 2023-09-29T22:22:47.668 DEBUG:boto:Token: None 2023-09-29T22:22:47.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa94dd5f1c53af46-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.710 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.710 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.710 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.711 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.711 DEBUG:boto:Method: GET 2023-09-29T22:22:47.711 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.711 DEBUG:boto:Data: 2023-09-29T22:22:47.711 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.711 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.711 DEBUG:boto:Params: {} 2023-09-29T22:22:47.711 DEBUG:boto:Token: None 2023-09-29T22:22:47.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba7d39247c72fe87-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.713 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.714 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.714 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.714 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.714 DEBUG:boto:Method: GET 2023-09-29T22:22:47.714 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.714 DEBUG:boto:Data: 2023-09-29T22:22:47.714 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.714 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.714 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.715 DEBUG:boto:Params: {} 2023-09-29T22:22:47.715 DEBUG:boto:Token: None 2023-09-29T22:22:47.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.715 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023c8d9e51f8fd332-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.717 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.718 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.718 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.718 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.718 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.718 DEBUG:boto:Method: GET 2023-09-29T22:22:47.718 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.719 DEBUG:boto:Data: 2023-09-29T22:22:47.719 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.719 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.719 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.719 DEBUG:boto:Params: {} 2023-09-29T22:22:47.719 DEBUG:boto:Token: None 2023-09-29T22:22:47.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f0613f2f400265-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.722 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.722 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.722 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.722 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.722 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.722 DEBUG:boto:Method: GET 2023-09-29T22:22:47.723 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.723 DEBUG:boto:Data: 2023-09-29T22:22:47.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.723 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.723 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.723 DEBUG:boto:Params: {} 2023-09-29T22:22:47.723 DEBUG:boto:Token: None 2023-09-29T22:22:47.723 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.723 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk= 2023-09-29T22:22:47.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk=', 'Content-Length': '0'} 2023-09-29T22:22:47.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbd1423b5f64658a-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.725 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.725 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.725 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.725 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.726 DEBUG:boto:Method: GET 2023-09-29T22:22:47.726 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.726 DEBUG:boto:Data: 2023-09-29T22:22:47.726 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.726 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.726 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.726 DEBUG:boto:Params: {} 2023-09-29T22:22:47.726 DEBUG:boto:Token: None 2023-09-29T22:22:47.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.727 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk= 2023-09-29T22:22:47.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk=', 'Content-Length': '0'} 2023-09-29T22:22:47.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbb76ac99ead0a21-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.729 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.729 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.729 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.729 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.729 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.729 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.730 DEBUG:boto:Data: 2023-09-29T22:22:47.730 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.730 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.730 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.730 DEBUG:boto:Params: {} 2023-09-29T22:22:47.730 DEBUG:boto:Token: None 2023-09-29T22:22:47.730 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.730 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uoAXr6zRXm7ckS3WNB74l1anLQE= 2023-09-29T22:22:47.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uoAXr6zRXm7ckS3WNB74l1anLQE=', 'Content-Length': '0'} 2023-09-29T22:22:47.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067ac5e302c077c75-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.732 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.733 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.733 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.733 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.733 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.733 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.733 DEBUG:boto:Data: 2023-09-29T22:22:47.733 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.733 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.733 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.733 DEBUG:boto:Params: {} 2023-09-29T22:22:47.734 DEBUG:boto:Token: None 2023-09-29T22:22:47.734 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.734 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uoAXr6zRXm7ckS3WNB74l1anLQE= 2023-09-29T22:22:47.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uoAXr6zRXm7ckS3WNB74l1anLQE=', 'Content-Length': '0'} 2023-09-29T22:22:47.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089e4fdf20a2696f0-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.736 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.736 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.736 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.736 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.736 DEBUG:boto:Method: GET 2023-09-29T22:22:47.736 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.736 DEBUG:boto:Data: 2023-09-29T22:22:47.737 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.737 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.737 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.737 DEBUG:boto:Params: {} 2023-09-29T22:22:47.737 DEBUG:boto:Token: None 2023-09-29T22:22:47.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk= 2023-09-29T22:22:47.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk=', 'Content-Length': '0'} 2023-09-29T22:22:47.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c3b37cc96ad7ddf-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.739 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.739 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.739 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.739 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.739 DEBUG:boto:Method: GET 2023-09-29T22:22:47.739 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.740 DEBUG:boto:Data: 2023-09-29T22:22:47.740 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.740 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.740 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.740 DEBUG:boto:Params: {} 2023-09-29T22:22:47.740 DEBUG:boto:Token: None 2023-09-29T22:22:47.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:47.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk= 2023-09-29T22:22:47.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2sNiPISryv84+hQxAnaa69fY7gk=', 'Content-Length': '0'} 2023-09-29T22:22:47.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9f88e3392a0d50b-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.742 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.742 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.742 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.742 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.743 DEBUG:boto:Method: GET 2023-09-29T22:22:47.743 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.743 DEBUG:boto:Data: 2023-09-29T22:22:47.743 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.743 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.743 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.743 DEBUG:boto:Params: {} 2023-09-29T22:22:47.743 DEBUG:boto:Token: None 2023-09-29T22:22:47.743 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.743 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ebf0e04079f998-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.746 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.746 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.746 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.746 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.746 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.746 DEBUG:boto:Method: GET 2023-09-29T22:22:47.747 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.747 DEBUG:boto:Data: 2023-09-29T22:22:47.747 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.747 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.747 DEBUG:boto:Params: {} 2023-09-29T22:22:47.747 DEBUG:boto:Token: None 2023-09-29T22:22:47.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.747 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383204a056d434d5-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.749 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.750 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.750 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.750 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.750 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.750 DEBUG:boto:Data: 2023-09-29T22:22:47.750 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.750 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.750 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.751 DEBUG:boto:Params: {} 2023-09-29T22:22:47.751 DEBUG:boto:Token: None 2023-09-29T22:22:47.751 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084deb3b3de572037-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.754 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.754 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.754 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.754 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.754 DEBUG:boto:Data: 2023-09-29T22:22:47.754 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.754 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.755 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.755 DEBUG:boto:Params: {} 2023-09-29T22:22:47.755 DEBUG:boto:Token: None 2023-09-29T22:22:47.755 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.755 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c312f792182e9ec9-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.758 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.758 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.758 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.758 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.758 DEBUG:boto:Method: GET 2023-09-29T22:22:47.758 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.759 DEBUG:boto:Data: 2023-09-29T22:22:47.759 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.759 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.759 DEBUG:boto:Params: {} 2023-09-29T22:22:47.759 DEBUG:boto:Token: None 2023-09-29T22:22:47.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.759 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb9ad0d0739ee1fc-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.761 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.761 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.761 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.761 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.762 DEBUG:boto:Method: GET 2023-09-29T22:22:47.762 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.762 DEBUG:boto:Data: 2023-09-29T22:22:47.762 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.762 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.762 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.762 DEBUG:boto:Params: {} 2023-09-29T22:22:47.762 DEBUG:boto:Token: None 2023-09-29T22:22:47.762 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.763 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e643a389ac406092-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.766 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.767 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.767 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.767 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.767 DEBUG:boto:Method: GET 2023-09-29T22:22:47.767 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.767 DEBUG:boto:Data: 2023-09-29T22:22:47.767 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.767 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.767 DEBUG:boto:Params: {} 2023-09-29T22:22:47.768 DEBUG:boto:Token: None 2023-09-29T22:22:47.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9c3dec42ddc223-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.770 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.771 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.771 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.771 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.771 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.771 DEBUG:boto:Method: GET 2023-09-29T22:22:47.771 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.771 DEBUG:boto:Data: 2023-09-29T22:22:47.772 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.772 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.772 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.772 DEBUG:boto:Params: {} 2023-09-29T22:22:47.772 DEBUG:boto:Token: None 2023-09-29T22:22:47.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092bf37cac4d563bf-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.774 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.775 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.775 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.775 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.775 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.775 DEBUG:boto:Method: GET 2023-09-29T22:22:47.775 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.776 DEBUG:boto:Data: 2023-09-29T22:22:47.776 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.776 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.776 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.776 DEBUG:boto:Params: {} 2023-09-29T22:22:47.776 DEBUG:boto:Token: None 2023-09-29T22:22:47.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.776 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo= 2023-09-29T22:22:47.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo=', 'Content-Length': '0'} 2023-09-29T22:22:47.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090cabdd399c079b2-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.818 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.818 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.819 DEBUG:boto:Method: GET 2023-09-29T22:22:47.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.819 DEBUG:boto:Data: 2023-09-29T22:22:47.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.819 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.819 DEBUG:boto:Params: {} 2023-09-29T22:22:47.820 DEBUG:boto:Token: None 2023-09-29T22:22:47.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo= 2023-09-29T22:22:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo=', 'Content-Length': '0'} 2023-09-29T22:22:47.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d99af7c933ac5ae-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.862 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.862 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.862 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.863 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.863 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.863 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.863 DEBUG:boto:Data: 2023-09-29T22:22:47.863 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.863 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.863 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.863 DEBUG:boto:Params: {} 2023-09-29T22:22:47.863 DEBUG:boto:Token: None 2023-09-29T22:22:47.863 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CGSPiakYEK/fuhFbBzg/t9pNCFA= 2023-09-29T22:22:47.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CGSPiakYEK/fuhFbBzg/t9pNCFA=', 'Content-Length': '0'} 2023-09-29T22:22:47.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ef1a7917a9b3c21-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.865 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.866 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.866 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.866 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.866 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.866 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.866 DEBUG:boto:Data: 2023-09-29T22:22:47.866 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.866 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.866 DEBUG:boto:Params: {} 2023-09-29T22:22:47.867 DEBUG:boto:Token: None 2023-09-29T22:22:47.867 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CGSPiakYEK/fuhFbBzg/t9pNCFA= 2023-09-29T22:22:47.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CGSPiakYEK/fuhFbBzg/t9pNCFA=', 'Content-Length': '0'} 2023-09-29T22:22:47.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1c90406812de50d-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.869 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.869 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.869 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.869 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.869 DEBUG:boto:Method: GET 2023-09-29T22:22:47.869 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.870 DEBUG:boto:Data: 2023-09-29T22:22:47.870 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.870 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.870 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.870 DEBUG:boto:Params: {} 2023-09-29T22:22:47.870 DEBUG:boto:Token: None 2023-09-29T22:22:47.870 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo= 2023-09-29T22:22:47.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo=', 'Content-Length': '0'} 2023-09-29T22:22:47.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ed88b8c5a01ef76-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.872 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.872 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.872 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.872 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.873 DEBUG:boto:Method: GET 2023-09-29T22:22:47.873 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.873 DEBUG:boto:Data: 2023-09-29T22:22:47.873 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.873 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.873 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.873 DEBUG:boto:Params: {} 2023-09-29T22:22:47.873 DEBUG:boto:Token: None 2023-09-29T22:22:47.873 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:47.874 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo= 2023-09-29T22:22:47.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:meLt8ri+MGe4TSOwSCrecFL3mzo=', 'Content-Length': '0'} 2023-09-29T22:22:47.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3a0c15fcf6f1d41-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.918 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.919 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.919 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.919 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.919 DEBUG:boto:Method: GET 2023-09-29T22:22:47.919 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.919 DEBUG:boto:Data: 2023-09-29T22:22:47.919 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.919 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.919 DEBUG:boto:Params: {} 2023-09-29T22:22:47.920 DEBUG:boto:Token: None 2023-09-29T22:22:47.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f99455ac8cbbdb80-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.922 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.923 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.923 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.923 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.923 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.923 DEBUG:boto:Method: GET 2023-09-29T22:22:47.923 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.923 DEBUG:boto:Data: 2023-09-29T22:22:47.924 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.924 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.924 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.924 DEBUG:boto:Params: {} 2023-09-29T22:22:47.924 DEBUG:boto:Token: None 2023-09-29T22:22:47.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc3d19d0ede2dbcf-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.927 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.927 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.927 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.927 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.927 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.927 DEBUG:boto:Data: 2023-09-29T22:22:47.927 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.927 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.928 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.928 DEBUG:boto:Params: {} 2023-09-29T22:22:47.928 DEBUG:boto:Token: None 2023-09-29T22:22:47.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089b827b00b8ce1ae-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.931 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.931 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.931 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.931 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:47.931 DEBUG:boto:Data: 2023-09-29T22:22:47.932 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.932 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.932 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.932 DEBUG:boto:Params: {} 2023-09-29T22:22:47.932 DEBUG:boto:Token: None 2023-09-29T22:22:47.932 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj 2023-09-29T22:22:47.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I= 2023-09-29T22:22:47.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UR7mIMDCtTHzmP1OaOqbftmDc/I=', 'Content-Length': '0'} 2023-09-29T22:22:47.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005847e545cf1c7962-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.935 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.935 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.935 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.935 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.936 DEBUG:boto:Method: GET 2023-09-29T22:22:47.936 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.936 DEBUG:boto:Data: 2023-09-29T22:22:47.936 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.936 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.936 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.936 DEBUG:boto:Params: {} 2023-09-29T22:22:47.936 DEBUG:boto:Token: None 2023-09-29T22:22:47.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086be7d506d32c086-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.938 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.938 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.939 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.939 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.939 DEBUG:boto:Method: GET 2023-09-29T22:22:47.939 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.939 DEBUG:boto:Data: 2023-09-29T22:22:47.939 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.939 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.939 DEBUG:boto:Params: {} 2023-09-29T22:22:47.939 DEBUG:boto:Token: None 2023-09-29T22:22:47.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:47.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM= 2023-09-29T22:22:47.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fc6+K0EHSQbizoYjSEnww3fFcfM=', 'Content-Length': '0'} 2023-09-29T22:22:47.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e519d8acf35ce624-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.982 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.982 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.983 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.983 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.983 DEBUG:boto:Method: GET 2023-09-29T22:22:47.983 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.983 DEBUG:boto:Data: 2023-09-29T22:22:47.983 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.983 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.983 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.983 DEBUG:boto:Params: {} 2023-09-29T22:22:47.983 DEBUG:boto:Token: None 2023-09-29T22:22:47.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.984 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000400789be5dab533-0065174e37-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.986 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.986 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:47.987 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:47.987 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.987 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.987 DEBUG:boto:Method: GET 2023-09-29T22:22:47.987 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:47.987 DEBUG:boto:Data: 2023-09-29T22:22:47.987 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.987 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.988 DEBUG:boto:Params: {} 2023-09-29T22:22:47.988 DEBUG:boto:Token: None 2023-09-29T22:22:47.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/ 2023-09-29T22:22:47.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ= 2023-09-29T22:22:47.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9p+pYwmv31gb7jFVwdtWUFb4xbQ=', 'Content-Length': '0'} 2023-09-29T22:22:47.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09fecf4447f1312-0065174e37-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.990 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:47.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:47.991 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.991 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.991 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.991 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.991 DEBUG:boto:Method: GET 2023-09-29T22:22:47.991 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.991 DEBUG:boto:Data: 2023-09-29T22:22:47.991 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.992 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.992 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.992 DEBUG:boto:Params: {} 2023-09-29T22:22:47.992 DEBUG:boto:Token: None 2023-09-29T22:22:47.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3SVOq8wp+SFXBERDKH+SxQUn4aI= 2023-09-29T22:22:47.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3SVOq8wp+SFXBERDKH+SxQUn4aI=', 'Content-Length': '0'} 2023-09-29T22:22:47.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008264f605db163f61-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.994 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.994 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.994 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.994 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.994 DEBUG:boto:Method: GET 2023-09-29T22:22:47.994 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.995 DEBUG:boto:Data: 2023-09-29T22:22:47.995 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:47.995 DEBUG:boto:Port: 8000 2023-09-29T22:22:47.995 DEBUG:boto:Params: {} 2023-09-29T22:22:47.995 DEBUG:boto:Token: None 2023-09-29T22:22:47.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.995 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3SVOq8wp+SFXBERDKH+SxQUn4aI= 2023-09-29T22:22:47.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3SVOq8wp+SFXBERDKH+SxQUn4aI=', 'Content-Length': '0'} 2023-09-29T22:22:47.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdef882a79eef264-0065174e37-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:47.997 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:47.997 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:47.997 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.997 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.998 DEBUG:boto:Method: HEAD 2023-09-29T22:22:47.998 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.998 DEBUG:boto:Data: 2023-09-29T22:22:47.998 DEBUG:boto:Headers: {} 2023-09-29T22:22:47.998 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:47.998 DEBUG:boto:Port: 8001 2023-09-29T22:22:47.998 DEBUG:boto:Params: {} 2023-09-29T22:22:47.998 DEBUG:boto:Token: None 2023-09-29T22:22:47.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:47 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:47.999 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mV3a/hyBDrGWUcgeamNr2tUpOjQ= 2023-09-29T22:22:47.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mV3a/hyBDrGWUcgeamNr2tUpOjQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000086b9d33808d15ac-0065174e37-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.000 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.000 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.000 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.001 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.001 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.001 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.001 DEBUG:boto:Data: 2023-09-29T22:22:48.001 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.001 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.001 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.001 DEBUG:boto:Params: {} 2023-09-29T22:22:48.001 DEBUG:boto:Token: None 2023-09-29T22:22:48.001 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.002 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YO4pHFZSycwyE6WU944kzs5lb4E= 2023-09-29T22:22:48.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YO4pHFZSycwyE6WU944kzs5lb4E=', 'Content-Length': '0'} 2023-09-29T22:22:48.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9c6e6b8bb92cc59-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.003 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.003 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.004 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.004 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.004 DEBUG:boto:Method: GET 2023-09-29T22:22:48.004 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.004 DEBUG:boto:Data: 2023-09-29T22:22:48.004 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.004 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.004 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.004 DEBUG:boto:Params: {} 2023-09-29T22:22:48.004 DEBUG:boto:Token: None 2023-09-29T22:22:48.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HvBVyj6R0ucHbGeHXW+sqIuBssQ= 2023-09-29T22:22:48.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HvBVyj6R0ucHbGeHXW+sqIuBssQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091ee479c1d87e852-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.007 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.007 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.007 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.007 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.007 DEBUG:boto:Method: GET 2023-09-29T22:22:48.007 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.007 DEBUG:boto:Data: 2023-09-29T22:22:48.007 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.007 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.007 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.008 DEBUG:boto:Params: {} 2023-09-29T22:22:48.008 DEBUG:boto:Token: None 2023-09-29T22:22:48.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:48.008 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HvBVyj6R0ucHbGeHXW+sqIuBssQ= 2023-09-29T22:22:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HvBVyj6R0ucHbGeHXW+sqIuBssQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a8c19427d9bafa-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.010 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.010 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.010 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.010 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.010 DEBUG:boto:Method: GET 2023-09-29T22:22:48.011 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.011 DEBUG:boto:Data: 2023-09-29T22:22:48.011 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.011 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.011 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.011 DEBUG:boto:Params: {} 2023-09-29T22:22:48.011 DEBUG:boto:Token: None 2023-09-29T22:22:48.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd7fb9c042fc54c6-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.058 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.059 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.059 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.059 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.059 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.059 DEBUG:boto:Method: GET 2023-09-29T22:22:48.059 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.059 DEBUG:boto:Data: 2023-09-29T22:22:48.059 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.059 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.060 DEBUG:boto:Params: {} 2023-09-29T22:22:48.060 DEBUG:boto:Token: None 2023-09-29T22:22:48.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e2c6c7ab45b0fa0-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.062 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.062 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.063 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.063 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.063 DEBUG:boto:Data: 2023-09-29T22:22:48.063 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.063 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.063 DEBUG:boto:Params: {} 2023-09-29T22:22:48.063 DEBUG:boto:Token: None 2023-09-29T22:22:48.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac975407d3c12089-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.067 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.067 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.067 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.067 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.067 DEBUG:boto:Data: 2023-09-29T22:22:48.067 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.067 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.067 DEBUG:boto:Params: {} 2023-09-29T22:22:48.067 DEBUG:boto:Token: None 2023-09-29T22:22:48.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4e3ca82fa7ced50-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.070 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.070 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.071 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.071 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.071 DEBUG:boto:Method: GET 2023-09-29T22:22:48.071 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.071 DEBUG:boto:Data: 2023-09-29T22:22:48.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.071 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.071 DEBUG:boto:Params: {} 2023-09-29T22:22:48.071 DEBUG:boto:Token: None 2023-09-29T22:22:48.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0fd8ba047c0f52a-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.074 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.074 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.074 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.074 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.074 DEBUG:boto:Method: GET 2023-09-29T22:22:48.074 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.074 DEBUG:boto:Data: 2023-09-29T22:22:48.074 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.074 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.074 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.075 DEBUG:boto:Params: {} 2023-09-29T22:22:48.075 DEBUG:boto:Token: None 2023-09-29T22:22:48.075 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f12b5face89d624-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.077 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.077 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.077 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.077 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.077 DEBUG:boto:Method: GET 2023-09-29T22:22:48.077 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.077 DEBUG:boto:Data: 2023-09-29T22:22:48.078 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.078 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.078 DEBUG:boto:Params: {} 2023-09-29T22:22:48.078 DEBUG:boto:Token: None 2023-09-29T22:22:48.078 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.078 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b39290ba68700acb-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.081 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.081 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.081 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.081 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.082 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.082 DEBUG:boto:Method: GET 2023-09-29T22:22:48.082 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.082 DEBUG:boto:Data: 2023-09-29T22:22:48.082 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.082 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.082 DEBUG:boto:Params: {} 2023-09-29T22:22:48.082 DEBUG:boto:Token: None 2023-09-29T22:22:48.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.083 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf44790b73bf44ea-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.085 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.085 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.085 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.086 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.086 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.086 DEBUG:boto:Method: GET 2023-09-29T22:22:48.086 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.086 DEBUG:boto:Data: 2023-09-29T22:22:48.086 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.086 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.086 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.086 DEBUG:boto:Params: {} 2023-09-29T22:22:48.086 DEBUG:boto:Token: None 2023-09-29T22:22:48.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc= 2023-09-29T22:22:48.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc=', 'Content-Length': '0'} 2023-09-29T22:22:48.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d19332c5e78b66b1-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.088 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.089 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.089 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.089 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.089 DEBUG:boto:Method: GET 2023-09-29T22:22:48.089 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.089 DEBUG:boto:Data: 2023-09-29T22:22:48.089 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.089 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.089 DEBUG:boto:Params: {} 2023-09-29T22:22:48.089 DEBUG:boto:Token: None 2023-09-29T22:22:48.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc= 2023-09-29T22:22:48.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc=', 'Content-Length': '0'} 2023-09-29T22:22:48.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026abd2349a3c4ba3-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.092 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.092 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.092 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.092 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.092 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.092 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.092 DEBUG:boto:Data: 2023-09-29T22:22:48.093 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.093 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.093 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.093 DEBUG:boto:Params: {} 2023-09-29T22:22:48.093 DEBUG:boto:Token: None 2023-09-29T22:22:48.093 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.093 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J/fXZNH7zyvAvTRjq65ahvMLrNI= 2023-09-29T22:22:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J/fXZNH7zyvAvTRjq65ahvMLrNI=', 'Content-Length': '0'} 2023-09-29T22:22:48.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a4d2089f1f2521b-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.095 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.095 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.095 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.095 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.095 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.095 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.095 DEBUG:boto:Data: 2023-09-29T22:22:48.096 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.096 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.096 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.096 DEBUG:boto:Params: {} 2023-09-29T22:22:48.096 DEBUG:boto:Token: None 2023-09-29T22:22:48.096 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J/fXZNH7zyvAvTRjq65ahvMLrNI= 2023-09-29T22:22:48.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J/fXZNH7zyvAvTRjq65ahvMLrNI=', 'Content-Length': '0'} 2023-09-29T22:22:48.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000207b58c602159fd9-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.098 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.098 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.098 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.098 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.098 DEBUG:boto:Method: GET 2023-09-29T22:22:48.099 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.099 DEBUG:boto:Data: 2023-09-29T22:22:48.099 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.099 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.099 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.099 DEBUG:boto:Params: {} 2023-09-29T22:22:48.099 DEBUG:boto:Token: None 2023-09-29T22:22:48.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.099 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc= 2023-09-29T22:22:48.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc=', 'Content-Length': '0'} 2023-09-29T22:22:48.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba5d805537b5a591-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.142 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.143 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.143 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.143 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.143 DEBUG:boto:Method: GET 2023-09-29T22:22:48.143 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.143 DEBUG:boto:Data: 2023-09-29T22:22:48.143 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.143 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.143 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.144 DEBUG:boto:Params: {} 2023-09-29T22:22:48.144 DEBUG:boto:Token: None 2023-09-29T22:22:48.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:48.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc= 2023-09-29T22:22:48.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AjOUekUBECTW/mtS8i0oWf06fpc=', 'Content-Length': '0'} 2023-09-29T22:22:48.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000596dd4c50da1b245-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.186 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.186 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.187 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.187 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.187 DEBUG:boto:Method: GET 2023-09-29T22:22:48.187 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.187 DEBUG:boto:Data: 2023-09-29T22:22:48.187 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.187 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.187 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.187 DEBUG:boto:Params: {} 2023-09-29T22:22:48.187 DEBUG:boto:Token: None 2023-09-29T22:22:48.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.188 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020a6ecc4db69a316-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.190 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.191 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.191 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.191 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.191 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.191 DEBUG:boto:Method: GET 2023-09-29T22:22:48.191 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.191 DEBUG:boto:Data: 2023-09-29T22:22:48.192 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.192 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.192 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.192 DEBUG:boto:Params: {} 2023-09-29T22:22:48.192 DEBUG:boto:Token: None 2023-09-29T22:22:48.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.192 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb4431b8872b9cb3-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.238 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.239 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.239 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.239 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.239 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.239 DEBUG:boto:Data: 2023-09-29T22:22:48.239 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.239 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.239 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.240 DEBUG:boto:Params: {} 2023-09-29T22:22:48.240 DEBUG:boto:Token: None 2023-09-29T22:22:48.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef2d8ed07343c13d-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.243 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.243 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.243 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.243 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.243 DEBUG:boto:Data: 2023-09-29T22:22:48.243 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.244 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.244 DEBUG:boto:Params: {} 2023-09-29T22:22:48.244 DEBUG:boto:Token: None 2023-09-29T22:22:48.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015745b663e6836c9-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.247 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.247 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.247 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.247 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.247 DEBUG:boto:Method: GET 2023-09-29T22:22:48.247 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.247 DEBUG:boto:Data: 2023-09-29T22:22:48.247 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.247 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.247 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.248 DEBUG:boto:Params: {} 2023-09-29T22:22:48.248 DEBUG:boto:Token: None 2023-09-29T22:22:48.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002552903a89ac49fd-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.250 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.250 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.250 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.250 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.250 DEBUG:boto:Method: GET 2023-09-29T22:22:48.250 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.250 DEBUG:boto:Data: 2023-09-29T22:22:48.250 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.251 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.251 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.251 DEBUG:boto:Params: {} 2023-09-29T22:22:48.251 DEBUG:boto:Token: None 2023-09-29T22:22:48.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.251 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046b4467ab02e5e07-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.253 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.253 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.254 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.254 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.254 DEBUG:boto:Method: GET 2023-09-29T22:22:48.254 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.254 DEBUG:boto:Data: 2023-09-29T22:22:48.254 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.254 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.254 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.254 DEBUG:boto:Params: {} 2023-09-29T22:22:48.254 DEBUG:boto:Token: None 2023-09-29T22:22:48.255 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.255 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc3f5730d419a07-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.257 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.257 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.258 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.258 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.258 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.258 DEBUG:boto:Method: GET 2023-09-29T22:22:48.258 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.258 DEBUG:boto:Data: 2023-09-29T22:22:48.258 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.258 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.258 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.258 DEBUG:boto:Params: {} 2023-09-29T22:22:48.259 DEBUG:boto:Token: None 2023-09-29T22:22:48.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.259 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038debe0a6ba55b0-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.261 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.261 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.262 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.262 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.262 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.262 DEBUG:boto:Method: GET 2023-09-29T22:22:48.262 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.262 DEBUG:boto:Data: 2023-09-29T22:22:48.262 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.262 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.262 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.262 DEBUG:boto:Params: {} 2023-09-29T22:22:48.263 DEBUG:boto:Token: None 2023-09-29T22:22:48.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.263 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4= 2023-09-29T22:22:48.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4=', 'Content-Length': '0'} 2023-09-29T22:22:48.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d8d3aa6f00cb3d-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.265 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.265 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.265 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.265 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.265 DEBUG:boto:Method: GET 2023-09-29T22:22:48.265 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.265 DEBUG:boto:Data: 2023-09-29T22:22:48.265 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.265 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.266 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.266 DEBUG:boto:Params: {} 2023-09-29T22:22:48.266 DEBUG:boto:Token: None 2023-09-29T22:22:48.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.266 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4= 2023-09-29T22:22:48.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4=', 'Content-Length': '0'} 2023-09-29T22:22:48.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7df2ee91b883844-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.268 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.268 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.268 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.268 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.268 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.268 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.269 DEBUG:boto:Data: 2023-09-29T22:22:48.269 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.269 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.269 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.269 DEBUG:boto:Params: {} 2023-09-29T22:22:48.269 DEBUG:boto:Token: None 2023-09-29T22:22:48.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.269 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8SqtcdLREYn4TLdldGlYFPjbiVs= 2023-09-29T22:22:48.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8SqtcdLREYn4TLdldGlYFPjbiVs=', 'Content-Length': '0'} 2023-09-29T22:22:48.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f588e626075bc6f9-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.271 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.271 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.271 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.271 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.271 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.272 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.272 DEBUG:boto:Data: 2023-09-29T22:22:48.272 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.272 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.272 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.272 DEBUG:boto:Params: {} 2023-09-29T22:22:48.272 DEBUG:boto:Token: None 2023-09-29T22:22:48.272 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8SqtcdLREYn4TLdldGlYFPjbiVs= 2023-09-29T22:22:48.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8SqtcdLREYn4TLdldGlYFPjbiVs=', 'Content-Length': '0'} 2023-09-29T22:22:48.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b0bb87f30cda548-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.274 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.274 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.274 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.274 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.274 DEBUG:boto:Method: GET 2023-09-29T22:22:48.275 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.275 DEBUG:boto:Data: 2023-09-29T22:22:48.275 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.275 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.275 DEBUG:boto:Params: {} 2023-09-29T22:22:48.275 DEBUG:boto:Token: None 2023-09-29T22:22:48.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4= 2023-09-29T22:22:48.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4=', 'Content-Length': '0'} 2023-09-29T22:22:48.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000418dd806888ccd9-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.277 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.278 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.278 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.278 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.278 DEBUG:boto:Method: GET 2023-09-29T22:22:48.278 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.278 DEBUG:boto:Data: 2023-09-29T22:22:48.278 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.278 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.278 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.278 DEBUG:boto:Params: {} 2023-09-29T22:22:48.278 DEBUG:boto:Token: None 2023-09-29T22:22:48.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:48.279 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4= 2023-09-29T22:22:48.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:359ULhdFUhQejflw0y684OcmNd4=', 'Content-Length': '0'} 2023-09-29T22:22:48.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d058af911b7d591c-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.322 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.322 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.322 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.323 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.323 DEBUG:boto:Method: GET 2023-09-29T22:22:48.323 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.323 DEBUG:boto:Data: 2023-09-29T22:22:48.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.323 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.323 DEBUG:boto:Params: {} 2023-09-29T22:22:48.323 DEBUG:boto:Token: None 2023-09-29T22:22:48.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4629d87fb20d71e-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.326 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.326 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.327 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.327 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.327 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.327 DEBUG:boto:Method: GET 2023-09-29T22:22:48.327 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.327 DEBUG:boto:Data: 2023-09-29T22:22:48.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.327 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.327 DEBUG:boto:Params: {} 2023-09-29T22:22:48.327 DEBUG:boto:Token: None 2023-09-29T22:22:48.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097237cd8d927884c-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.370 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.370 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.371 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.371 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.371 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.371 DEBUG:boto:Data: 2023-09-29T22:22:48.371 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.371 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.371 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.371 DEBUG:boto:Params: {} 2023-09-29T22:22:48.372 DEBUG:boto:Token: None 2023-09-29T22:22:48.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d32fa8826f04e655-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.375 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.375 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.375 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.375 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.375 DEBUG:boto:Data: 2023-09-29T22:22:48.375 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.375 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.375 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.375 DEBUG:boto:Params: {} 2023-09-29T22:22:48.377 DEBUG:boto:Token: None 2023-09-29T22:22:48.377 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.377 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6c0900f1d64be1a-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.380 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.380 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.380 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.381 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.381 DEBUG:boto:Method: GET 2023-09-29T22:22:48.381 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.381 DEBUG:boto:Data: 2023-09-29T22:22:48.381 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.381 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.381 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.381 DEBUG:boto:Params: {} 2023-09-29T22:22:48.381 DEBUG:boto:Token: None 2023-09-29T22:22:48.382 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.382 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000180f5f9fd260de94-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.426 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.426 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.427 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.427 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.427 DEBUG:boto:Method: GET 2023-09-29T22:22:48.427 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.427 DEBUG:boto:Data: 2023-09-29T22:22:48.427 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.427 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.427 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.427 DEBUG:boto:Params: {} 2023-09-29T22:22:48.427 DEBUG:boto:Token: None 2023-09-29T22:22:48.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc3afc1f38141fef-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.430 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.430 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.430 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.430 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.430 DEBUG:boto:Method: GET 2023-09-29T22:22:48.430 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.430 DEBUG:boto:Data: 2023-09-29T22:22:48.431 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.431 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.431 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.431 DEBUG:boto:Params: {} 2023-09-29T22:22:48.431 DEBUG:boto:Token: None 2023-09-29T22:22:48.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.431 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31cdfb90cd71b63-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.434 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.434 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.434 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.434 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.434 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.434 DEBUG:boto:Method: GET 2023-09-29T22:22:48.435 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.435 DEBUG:boto:Data: 2023-09-29T22:22:48.435 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.435 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.435 DEBUG:boto:Params: {} 2023-09-29T22:22:48.435 DEBUG:boto:Token: None 2023-09-29T22:22:48.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.435 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f557d30edeb1f7-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.438 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.438 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.438 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.438 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.438 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.439 DEBUG:boto:Method: GET 2023-09-29T22:22:48.439 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.439 DEBUG:boto:Data: 2023-09-29T22:22:48.439 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.439 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.439 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.439 DEBUG:boto:Params: {} 2023-09-29T22:22:48.439 DEBUG:boto:Token: None 2023-09-29T22:22:48.439 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM= 2023-09-29T22:22:48.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM=', 'Content-Length': '0'} 2023-09-29T22:22:48.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077fa3821390f217e-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.441 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.442 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.442 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.442 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.442 DEBUG:boto:Method: GET 2023-09-29T22:22:48.442 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.442 DEBUG:boto:Data: 2023-09-29T22:22:48.442 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.442 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.442 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.443 DEBUG:boto:Params: {} 2023-09-29T22:22:48.443 DEBUG:boto:Token: None 2023-09-29T22:22:48.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.443 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM= 2023-09-29T22:22:48.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM=', 'Content-Length': '0'} 2023-09-29T22:22:48.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbc72de23410f8c6-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.486 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.486 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.487 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.487 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.487 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.487 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.487 DEBUG:boto:Data: 2023-09-29T22:22:48.487 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.487 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.487 DEBUG:boto:Params: {} 2023-09-29T22:22:48.487 DEBUG:boto:Token: None 2023-09-29T22:22:48.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NkLllAowmgIMXDVATH1F9f4eyhk= 2023-09-29T22:22:48.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NkLllAowmgIMXDVATH1F9f4eyhk=', 'Content-Length': '0'} 2023-09-29T22:22:48.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003137fab85f7b989d-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.490 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.490 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.490 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.490 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.490 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.490 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.490 DEBUG:boto:Data: 2023-09-29T22:22:48.491 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.491 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.491 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.491 DEBUG:boto:Params: {} 2023-09-29T22:22:48.491 DEBUG:boto:Token: None 2023-09-29T22:22:48.491 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NkLllAowmgIMXDVATH1F9f4eyhk= 2023-09-29T22:22:48.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NkLllAowmgIMXDVATH1F9f4eyhk=', 'Content-Length': '0'} 2023-09-29T22:22:48.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a38c27bc7edba6ef-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.493 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.493 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.494 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.494 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.494 DEBUG:boto:Method: GET 2023-09-29T22:22:48.494 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.494 DEBUG:boto:Data: 2023-09-29T22:22:48.494 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.494 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.494 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.494 DEBUG:boto:Params: {} 2023-09-29T22:22:48.495 DEBUG:boto:Token: None 2023-09-29T22:22:48.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM= 2023-09-29T22:22:48.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM=', 'Content-Length': '0'} 2023-09-29T22:22:48.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d095a263dd5d8359-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.538 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.539 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.539 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.539 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.539 DEBUG:boto:Method: GET 2023-09-29T22:22:48.539 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.539 DEBUG:boto:Data: 2023-09-29T22:22:48.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.540 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.540 DEBUG:boto:Params: {} 2023-09-29T22:22:48.540 DEBUG:boto:Token: None 2023-09-29T22:22:48.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:48.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM= 2023-09-29T22:22:48.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qiHEJUQi5qM+I+bbIgUol5QIOYM=', 'Content-Length': '0'} 2023-09-29T22:22:48.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000195f08b8f6ef8c26-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.582 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.582 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.582 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.583 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.583 DEBUG:boto:Method: GET 2023-09-29T22:22:48.583 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.583 DEBUG:boto:Data: 2023-09-29T22:22:48.583 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.583 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.583 DEBUG:boto:Params: {} 2023-09-29T22:22:48.583 DEBUG:boto:Token: None 2023-09-29T22:22:48.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6df174b656c1d46-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.586 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.586 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.587 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.587 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.587 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.587 DEBUG:boto:Method: GET 2023-09-29T22:22:48.587 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.587 DEBUG:boto:Data: 2023-09-29T22:22:48.587 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.587 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.587 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.588 DEBUG:boto:Params: {} 2023-09-29T22:22:48.588 DEBUG:boto:Token: None 2023-09-29T22:22:48.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873dca69b672d542-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.634 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.635 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.635 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.635 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.635 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.635 DEBUG:boto:Data: 2023-09-29T22:22:48.635 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.635 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.635 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.635 DEBUG:boto:Params: {} 2023-09-29T22:22:48.636 DEBUG:boto:Token: None 2023-09-29T22:22:48.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae394ab52dccf737-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.639 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.639 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.639 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.639 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.639 DEBUG:boto:Data: 2023-09-29T22:22:48.639 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.639 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.639 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.639 DEBUG:boto:Params: {} 2023-09-29T22:22:48.640 DEBUG:boto:Token: None 2023-09-29T22:22:48.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074c9af9bf72b5721-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.643 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.643 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.643 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.643 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.643 DEBUG:boto:Method: GET 2023-09-29T22:22:48.643 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.643 DEBUG:boto:Data: 2023-09-29T22:22:48.643 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.643 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.644 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.644 DEBUG:boto:Params: {} 2023-09-29T22:22:48.644 DEBUG:boto:Token: None 2023-09-29T22:22:48.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a01ac76c7bcc9aa1-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.686 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.686 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.687 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.687 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.687 DEBUG:boto:Method: GET 2023-09-29T22:22:48.687 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.687 DEBUG:boto:Data: 2023-09-29T22:22:48.687 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.687 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.687 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.687 DEBUG:boto:Params: {} 2023-09-29T22:22:48.687 DEBUG:boto:Token: None 2023-09-29T22:22:48.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.688 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d90eee93e26a21ea-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.690 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.690 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.690 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.690 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.691 DEBUG:boto:Method: GET 2023-09-29T22:22:48.691 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.691 DEBUG:boto:Data: 2023-09-29T22:22:48.691 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.691 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.691 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.691 DEBUG:boto:Params: {} 2023-09-29T22:22:48.691 DEBUG:boto:Token: None 2023-09-29T22:22:48.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.692 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca451365d0fc2e0-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.694 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.694 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.694 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.695 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.695 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.695 DEBUG:boto:Method: GET 2023-09-29T22:22:48.695 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.695 DEBUG:boto:Data: 2023-09-29T22:22:48.695 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.695 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.695 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.695 DEBUG:boto:Params: {} 2023-09-29T22:22:48.695 DEBUG:boto:Token: None 2023-09-29T22:22:48.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.696 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065893c56138d2340-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.698 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.698 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.699 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.699 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.699 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.699 DEBUG:boto:Method: GET 2023-09-29T22:22:48.699 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.699 DEBUG:boto:Data: 2023-09-29T22:22:48.699 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.699 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.699 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.699 DEBUG:boto:Params: {} 2023-09-29T22:22:48.699 DEBUG:boto:Token: None 2023-09-29T22:22:48.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30= 2023-09-29T22:22:48.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30=', 'Content-Length': '0'} 2023-09-29T22:22:48.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b023b8c631e37d18-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.702 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.702 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.702 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.702 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.702 DEBUG:boto:Method: GET 2023-09-29T22:22:48.702 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.703 DEBUG:boto:Data: 2023-09-29T22:22:48.703 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.703 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.703 DEBUG:boto:Params: {} 2023-09-29T22:22:48.703 DEBUG:boto:Token: None 2023-09-29T22:22:48.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.703 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30= 2023-09-29T22:22:48.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30=', 'Content-Length': '0'} 2023-09-29T22:22:48.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e0b16d4dbe3f909-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.705 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.705 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.705 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.705 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.705 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.706 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.706 DEBUG:boto:Data: 2023-09-29T22:22:48.706 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.706 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.706 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.706 DEBUG:boto:Params: {} 2023-09-29T22:22:48.706 DEBUG:boto:Token: None 2023-09-29T22:22:48.706 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.706 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w9Wm/3jkHzsqdB2OInf8cvY1LiE= 2023-09-29T22:22:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w9Wm/3jkHzsqdB2OInf8cvY1LiE=', 'Content-Length': '0'} 2023-09-29T22:22:48.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adc9ba3af69ec581-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.708 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.708 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.708 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.708 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.709 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.709 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.709 DEBUG:boto:Data: 2023-09-29T22:22:48.709 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.709 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.709 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.709 DEBUG:boto:Params: {} 2023-09-29T22:22:48.709 DEBUG:boto:Token: None 2023-09-29T22:22:48.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w9Wm/3jkHzsqdB2OInf8cvY1LiE= 2023-09-29T22:22:48.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w9Wm/3jkHzsqdB2OInf8cvY1LiE=', 'Content-Length': '0'} 2023-09-29T22:22:48.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b509b15bca921f7b-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.712 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.712 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.712 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.712 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.712 DEBUG:boto:Method: GET 2023-09-29T22:22:48.712 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.712 DEBUG:boto:Data: 2023-09-29T22:22:48.712 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.713 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.713 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.713 DEBUG:boto:Params: {} 2023-09-29T22:22:48.713 DEBUG:boto:Token: None 2023-09-29T22:22:48.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.713 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30= 2023-09-29T22:22:48.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30=', 'Content-Length': '0'} 2023-09-29T22:22:48.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c9b9925e45b232a-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.758 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.759 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.759 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.759 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.759 DEBUG:boto:Method: GET 2023-09-29T22:22:48.759 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.759 DEBUG:boto:Data: 2023-09-29T22:22:48.759 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.759 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.759 DEBUG:boto:Params: {} 2023-09-29T22:22:48.760 DEBUG:boto:Token: None 2023-09-29T22:22:48.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:48.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30= 2023-09-29T22:22:48.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5wjszJdYzkepNVEarvDqU+SFv30=', 'Content-Length': '0'} 2023-09-29T22:22:48.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cddbf3bb0f60f23-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.762 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.762 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.762 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.762 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.762 DEBUG:boto:Method: GET 2023-09-29T22:22:48.762 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.762 DEBUG:boto:Data: 2023-09-29T22:22:48.762 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.762 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.763 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.763 DEBUG:boto:Params: {} 2023-09-29T22:22:48.763 DEBUG:boto:Token: None 2023-09-29T22:22:48.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.763 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cbb97d34cec835a-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.765 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.766 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.766 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.766 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.766 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.766 DEBUG:boto:Method: GET 2023-09-29T22:22:48.766 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.766 DEBUG:boto:Data: 2023-09-29T22:22:48.766 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.767 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.767 DEBUG:boto:Params: {} 2023-09-29T22:22:48.767 DEBUG:boto:Token: None 2023-09-29T22:22:48.767 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.767 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136eac7ec0432f4b-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.770 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.770 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.770 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.770 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.770 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.771 DEBUG:boto:Data: 2023-09-29T22:22:48.771 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.771 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.771 DEBUG:boto:Params: {} 2023-09-29T22:22:48.771 DEBUG:boto:Token: None 2023-09-29T22:22:48.771 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001582e827b4a3eb47-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.774 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.774 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.774 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.775 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.775 DEBUG:boto:Data: 2023-09-29T22:22:48.775 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.775 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.775 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.775 DEBUG:boto:Params: {} 2023-09-29T22:22:48.775 DEBUG:boto:Token: None 2023-09-29T22:22:48.775 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.775 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eba1a327533a23c0-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.778 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.778 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.779 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.779 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.779 DEBUG:boto:Method: GET 2023-09-29T22:22:48.779 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.779 DEBUG:boto:Data: 2023-09-29T22:22:48.779 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.779 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.779 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.779 DEBUG:boto:Params: {} 2023-09-29T22:22:48.779 DEBUG:boto:Token: None 2023-09-29T22:22:48.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4f05153c744a23-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.822 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.822 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.823 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.823 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.823 DEBUG:boto:Method: GET 2023-09-29T22:22:48.823 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.823 DEBUG:boto:Data: 2023-09-29T22:22:48.823 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.823 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.823 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.823 DEBUG:boto:Params: {} 2023-09-29T22:22:48.823 DEBUG:boto:Token: None 2023-09-29T22:22:48.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e02e6b9f8e8172b1-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.826 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.826 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.826 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.826 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.826 DEBUG:boto:Method: GET 2023-09-29T22:22:48.826 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.826 DEBUG:boto:Data: 2023-09-29T22:22:48.826 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.827 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.827 DEBUG:boto:Params: {} 2023-09-29T22:22:48.827 DEBUG:boto:Token: None 2023-09-29T22:22:48.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0daf37cc9e66dcb-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.870 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.871 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.871 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.871 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.871 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.871 DEBUG:boto:Method: GET 2023-09-29T22:22:48.871 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.871 DEBUG:boto:Data: 2023-09-29T22:22:48.871 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.871 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.871 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.872 DEBUG:boto:Params: {} 2023-09-29T22:22:48.872 DEBUG:boto:Token: None 2023-09-29T22:22:48.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6b13db6e1e734a-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.875 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.875 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.875 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.875 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.875 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.876 DEBUG:boto:Method: GET 2023-09-29T22:22:48.876 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.876 DEBUG:boto:Data: 2023-09-29T22:22:48.876 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.876 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.876 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.876 DEBUG:boto:Params: {} 2023-09-29T22:22:48.876 DEBUG:boto:Token: None 2023-09-29T22:22:48.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA= 2023-09-29T22:22:48.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA=', 'Content-Length': '0'} 2023-09-29T22:22:48.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a0cc29e0f49853a-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.878 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.878 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.879 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.879 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.879 DEBUG:boto:Method: GET 2023-09-29T22:22:48.879 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.879 DEBUG:boto:Data: 2023-09-29T22:22:48.879 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.879 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.879 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.879 DEBUG:boto:Params: {} 2023-09-29T22:22:48.879 DEBUG:boto:Token: None 2023-09-29T22:22:48.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA= 2023-09-29T22:22:48.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA=', 'Content-Length': '0'} 2023-09-29T22:22:48.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbbe30e438022f61-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.882 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.882 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.882 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.882 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.882 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.882 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.882 DEBUG:boto:Data: 2023-09-29T22:22:48.882 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.882 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.882 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.883 DEBUG:boto:Params: {} 2023-09-29T22:22:48.883 DEBUG:boto:Token: None 2023-09-29T22:22:48.883 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.883 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sEdkGuWMvnA5lGb4wCprpvFRePo= 2023-09-29T22:22:48.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sEdkGuWMvnA5lGb4wCprpvFRePo=', 'Content-Length': '0'} 2023-09-29T22:22:48.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000225f93cfd8448f08-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.885 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.885 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.885 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.885 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.885 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.885 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.885 DEBUG:boto:Data: 2023-09-29T22:22:48.885 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.885 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.886 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.886 DEBUG:boto:Params: {} 2023-09-29T22:22:48.886 DEBUG:boto:Token: None 2023-09-29T22:22:48.886 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.886 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sEdkGuWMvnA5lGb4wCprpvFRePo= 2023-09-29T22:22:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sEdkGuWMvnA5lGb4wCprpvFRePo=', 'Content-Length': '0'} 2023-09-29T22:22:48.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097dc3d3899000324-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.888 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.888 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.888 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.888 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.888 DEBUG:boto:Method: GET 2023-09-29T22:22:48.888 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.888 DEBUG:boto:Data: 2023-09-29T22:22:48.889 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.889 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.889 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.889 DEBUG:boto:Params: {} 2023-09-29T22:22:48.889 DEBUG:boto:Token: None 2023-09-29T22:22:48.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.889 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA= 2023-09-29T22:22:48.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA=', 'Content-Length': '0'} 2023-09-29T22:22:48.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ce5fdbc0b53f23a-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.891 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.891 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.891 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.891 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.891 DEBUG:boto:Method: GET 2023-09-29T22:22:48.892 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.892 DEBUG:boto:Data: 2023-09-29T22:22:48.892 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.892 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.892 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.892 DEBUG:boto:Params: {} 2023-09-29T22:22:48.892 DEBUG:boto:Token: None 2023-09-29T22:22:48.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:48.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA= 2023-09-29T22:22:48.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xoLadaIq0vaUwNYZn7Ipp5YUAFA=', 'Content-Length': '0'} 2023-09-29T22:22:48.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4e6472f010056eb-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.934 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.934 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.935 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.935 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.935 DEBUG:boto:Method: GET 2023-09-29T22:22:48.935 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.935 DEBUG:boto:Data: 2023-09-29T22:22:48.935 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.935 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.935 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.935 DEBUG:boto:Params: {} 2023-09-29T22:22:48.935 DEBUG:boto:Token: None 2023-09-29T22:22:48.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eecea264c4f1ca7-0065174e38-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.938 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.939 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:48.939 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:48.939 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.939 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.939 DEBUG:boto:Method: GET 2023-09-29T22:22:48.939 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:48.939 DEBUG:boto:Data: 2023-09-29T22:22:48.939 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.939 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.940 DEBUG:boto:Params: {} 2023-09-29T22:22:48.940 DEBUG:boto:Token: None 2023-09-29T22:22:48.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/ 2023-09-29T22:22:48.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc= 2023-09-29T22:22:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CH7YMDsthqavWpB8nh8f+l3+Grc=', 'Content-Length': '0'} 2023-09-29T22:22:48.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4522bc92c2c4fd8-0065174e38-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.986 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:48.986 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.987 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.987 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.987 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.987 DEBUG:boto:Data: 2023-09-29T22:22:48.987 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.987 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.987 DEBUG:boto:Params: {} 2023-09-29T22:22:48.987 DEBUG:boto:Token: None 2023-09-29T22:22:48.988 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df27f97cf63c8a85-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.991 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.991 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.991 DEBUG:boto:Method: HEAD 2023-09-29T22:22:48.991 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:48.991 DEBUG:boto:Data: 2023-09-29T22:22:48.991 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.992 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:48.992 DEBUG:boto:Port: 8000 2023-09-29T22:22:48.992 DEBUG:boto:Params: {} 2023-09-29T22:22:48.992 DEBUG:boto:Token: None 2023-09-29T22:22:48.992 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj 2023-09-29T22:22:48.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs= 2023-09-29T22:22:48.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BVDMUY7Mobv6c6qIqbYAYkT+hAs=', 'Content-Length': '0'} 2023-09-29T22:22:48.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000709235295b795635-0065174e38-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:48.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:48.995 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:48.995 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:48.995 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.995 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.995 DEBUG:boto:Method: GET 2023-09-29T22:22:48.995 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.996 DEBUG:boto:Data: 2023-09-29T22:22:48.996 DEBUG:boto:Headers: {} 2023-09-29T22:22:48.996 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:48.996 DEBUG:boto:Port: 8001 2023-09-29T22:22:48.996 DEBUG:boto:Params: {} 2023-09-29T22:22:48.996 DEBUG:boto:Token: None 2023-09-29T22:22:48.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:48 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:48.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ= 2023-09-29T22:22:48.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:k0qfWYOZViyFFk+GxqPSZO4n8aQ=', 'Content-Length': '0'} 2023-09-29T22:22:48.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000110289ae37302964-0065174e38-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.038 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.038 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.038 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.039 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.039 DEBUG:boto:Method: GET 2023-09-29T22:22:49.039 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.039 DEBUG:boto:Data: 2023-09-29T22:22:49.039 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.039 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.039 DEBUG:boto:Params: {} 2023-09-29T22:22:49.039 DEBUG:boto:Token: None 2023-09-29T22:22:49.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebde3247e1f924e3-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.041 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.042 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.042 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.042 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.042 DEBUG:boto:Method: GET 2023-09-29T22:22:49.042 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.042 DEBUG:boto:Data: 2023-09-29T22:22:49.042 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.042 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.042 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.042 DEBUG:boto:Params: {} 2023-09-29T22:22:49.043 DEBUG:boto:Token: None 2023-09-29T22:22:49.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7462f59c4883bbb-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.045 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.045 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.046 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.046 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.046 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.046 DEBUG:boto:Method: GET 2023-09-29T22:22:49.046 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.046 DEBUG:boto:Data: 2023-09-29T22:22:49.046 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.046 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.046 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.046 DEBUG:boto:Params: {} 2023-09-29T22:22:49.047 DEBUG:boto:Token: None 2023-09-29T22:22:49.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a01d2bf736a0b6-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.049 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.050 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.050 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.050 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.050 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.050 DEBUG:boto:Method: GET 2023-09-29T22:22:49.050 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.050 DEBUG:boto:Data: 2023-09-29T22:22:49.050 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.050 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.051 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.051 DEBUG:boto:Params: {} 2023-09-29T22:22:49.051 DEBUG:boto:Token: None 2023-09-29T22:22:49.051 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.051 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ= 2023-09-29T22:22:49.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ=', 'Content-Length': '0'} 2023-09-29T22:22:49.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a61e54ea8a3b42de-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.053 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.053 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.053 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.053 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.053 DEBUG:boto:Method: GET 2023-09-29T22:22:49.053 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.054 DEBUG:boto:Data: 2023-09-29T22:22:49.054 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.054 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.054 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.054 DEBUG:boto:Params: {} 2023-09-29T22:22:49.054 DEBUG:boto:Token: None 2023-09-29T22:22:49.054 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.054 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ= 2023-09-29T22:22:49.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ=', 'Content-Length': '0'} 2023-09-29T22:22:49.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a1f47a58eb6cff8-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.056 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.056 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.056 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.056 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.056 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.057 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.057 DEBUG:boto:Data: 2023-09-29T22:22:49.057 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.057 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.057 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.057 DEBUG:boto:Params: {} 2023-09-29T22:22:49.057 DEBUG:boto:Token: None 2023-09-29T22:22:49.057 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.057 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fePnF1uyTceZ2iLsSLA00Pr5jm0= 2023-09-29T22:22:49.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fePnF1uyTceZ2iLsSLA00Pr5jm0=', 'Content-Length': '0'} 2023-09-29T22:22:49.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcfa859392c8cbf0-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.059 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.060 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.060 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.060 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.060 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.060 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.060 DEBUG:boto:Data: 2023-09-29T22:22:49.060 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.060 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.060 DEBUG:boto:Params: {} 2023-09-29T22:22:49.061 DEBUG:boto:Token: None 2023-09-29T22:22:49.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fePnF1uyTceZ2iLsSLA00Pr5jm0= 2023-09-29T22:22:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fePnF1uyTceZ2iLsSLA00Pr5jm0=', 'Content-Length': '0'} 2023-09-29T22:22:49.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030e8ceadc50f7dc3-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.063 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.063 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.063 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.063 DEBUG:boto:Method: GET 2023-09-29T22:22:49.064 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.064 DEBUG:boto:Data: 2023-09-29T22:22:49.064 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.064 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.064 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.064 DEBUG:boto:Params: {} 2023-09-29T22:22:49.064 DEBUG:boto:Token: None 2023-09-29T22:22:49.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ= 2023-09-29T22:22:49.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ=', 'Content-Length': '0'} 2023-09-29T22:22:49.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000353dbd8dfbd92ef4-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.106 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.106 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.106 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.107 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.107 DEBUG:boto:Method: GET 2023-09-29T22:22:49.107 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.107 DEBUG:boto:Data: 2023-09-29T22:22:49.107 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.107 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.107 DEBUG:boto:Params: {} 2023-09-29T22:22:49.107 DEBUG:boto:Token: None 2023-09-29T22:22:49.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:49.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ= 2023-09-29T22:22:49.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylnYzLYyzlHiMb64I2614ve6wlQ=', 'Content-Length': '0'} 2023-09-29T22:22:49.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034c95b35cbe1b90d-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.110 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.110 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.110 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.110 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.110 DEBUG:boto:Method: GET 2023-09-29T22:22:49.110 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.110 DEBUG:boto:Data: 2023-09-29T22:22:49.110 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.111 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.111 DEBUG:boto:Params: {} 2023-09-29T22:22:49.111 DEBUG:boto:Token: None 2023-09-29T22:22:49.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad113c8a58f91e3f-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.154 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.155 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.155 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.155 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.155 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.155 DEBUG:boto:Method: GET 2023-09-29T22:22:49.155 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.155 DEBUG:boto:Data: 2023-09-29T22:22:49.155 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.156 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.156 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.156 DEBUG:boto:Params: {} 2023-09-29T22:22:49.156 DEBUG:boto:Token: None 2023-09-29T22:22:49.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.156 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713903eae77e1f4f-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.158 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.159 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.159 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.159 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.159 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.159 DEBUG:boto:Data: 2023-09-29T22:22:49.159 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.159 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.159 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.160 DEBUG:boto:Params: {} 2023-09-29T22:22:49.160 DEBUG:boto:Token: None 2023-09-29T22:22:49.160 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.160 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091e8e5d6daf1d65b-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.163 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.163 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.163 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.163 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.163 DEBUG:boto:Data: 2023-09-29T22:22:49.163 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.163 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.164 DEBUG:boto:Params: {} 2023-09-29T22:22:49.164 DEBUG:boto:Token: None 2023-09-29T22:22:49.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.164 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004829cdef407824b3-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.167 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.167 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.167 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.167 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.167 DEBUG:boto:Method: GET 2023-09-29T22:22:49.167 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.167 DEBUG:boto:Data: 2023-09-29T22:22:49.167 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.168 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.168 DEBUG:boto:Params: {} 2023-09-29T22:22:49.168 DEBUG:boto:Token: None 2023-09-29T22:22:49.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001add3825f7ae0a30-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.214 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.214 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.214 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.215 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.215 DEBUG:boto:Method: GET 2023-09-29T22:22:49.215 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.215 DEBUG:boto:Data: 2023-09-29T22:22:49.215 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.215 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.215 DEBUG:boto:Params: {} 2023-09-29T22:22:49.215 DEBUG:boto:Token: None 2023-09-29T22:22:49.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.216 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017a5edffbec125e8-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.218 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.218 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.218 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.218 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.218 DEBUG:boto:Method: GET 2023-09-29T22:22:49.218 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.218 DEBUG:boto:Data: 2023-09-29T22:22:49.218 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.218 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.219 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.219 DEBUG:boto:Params: {} 2023-09-29T22:22:49.219 DEBUG:boto:Token: None 2023-09-29T22:22:49.219 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.219 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001769f91f7f1d7b45-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.262 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.262 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.263 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.263 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.263 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.263 DEBUG:boto:Method: GET 2023-09-29T22:22:49.263 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.263 DEBUG:boto:Data: 2023-09-29T22:22:49.263 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.263 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.263 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.263 DEBUG:boto:Params: {} 2023-09-29T22:22:49.264 DEBUG:boto:Token: None 2023-09-29T22:22:49.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff43a1356527f37-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.266 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.267 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.267 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.267 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.267 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.267 DEBUG:boto:Method: GET 2023-09-29T22:22:49.267 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.267 DEBUG:boto:Data: 2023-09-29T22:22:49.268 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.268 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.268 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.268 DEBUG:boto:Params: {} 2023-09-29T22:22:49.268 DEBUG:boto:Token: None 2023-09-29T22:22:49.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM= 2023-09-29T22:22:49.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM=', 'Content-Length': '0'} 2023-09-29T22:22:49.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ca1d12a94cfd673-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.270 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.270 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.270 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.271 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.271 DEBUG:boto:Method: GET 2023-09-29T22:22:49.271 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.271 DEBUG:boto:Data: 2023-09-29T22:22:49.271 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.271 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.271 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.271 DEBUG:boto:Params: {} 2023-09-29T22:22:49.271 DEBUG:boto:Token: None 2023-09-29T22:22:49.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM= 2023-09-29T22:22:49.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM=', 'Content-Length': '0'} 2023-09-29T22:22:49.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac2559e7dfd072d4-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.318 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.318 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.319 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.319 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.319 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.319 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.319 DEBUG:boto:Data: 2023-09-29T22:22:49.319 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.319 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.319 DEBUG:boto:Params: {} 2023-09-29T22:22:49.319 DEBUG:boto:Token: None 2023-09-29T22:22:49.320 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9fpunLrS93Z3+QJVn0HH6fr/VJI= 2023-09-29T22:22:49.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9fpunLrS93Z3+QJVn0HH6fr/VJI=', 'Content-Length': '0'} 2023-09-29T22:22:49.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030ab6b86c993df3e-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.321 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.322 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.322 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.322 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.322 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.322 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.322 DEBUG:boto:Data: 2023-09-29T22:22:49.322 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.322 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.322 DEBUG:boto:Params: {} 2023-09-29T22:22:49.323 DEBUG:boto:Token: None 2023-09-29T22:22:49.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9fpunLrS93Z3+QJVn0HH6fr/VJI= 2023-09-29T22:22:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9fpunLrS93Z3+QJVn0HH6fr/VJI=', 'Content-Length': '0'} 2023-09-29T22:22:49.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084be29e4ae6a5d40-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.325 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.325 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.325 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.325 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.325 DEBUG:boto:Method: GET 2023-09-29T22:22:49.325 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.325 DEBUG:boto:Data: 2023-09-29T22:22:49.325 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.325 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.325 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.326 DEBUG:boto:Params: {} 2023-09-29T22:22:49.326 DEBUG:boto:Token: None 2023-09-29T22:22:49.326 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.326 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM= 2023-09-29T22:22:49.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM=', 'Content-Length': '0'} 2023-09-29T22:22:49.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cbf1bd67a7d1e3a-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.370 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.370 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.371 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.371 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.371 DEBUG:boto:Method: GET 2023-09-29T22:22:49.371 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.371 DEBUG:boto:Data: 2023-09-29T22:22:49.371 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.371 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.371 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.371 DEBUG:boto:Params: {} 2023-09-29T22:22:49.371 DEBUG:boto:Token: None 2023-09-29T22:22:49.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:49.372 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM= 2023-09-29T22:22:49.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PdlnKBYaFObVGIRsSN3L7SJaFaM=', 'Content-Length': '0'} 2023-09-29T22:22:49.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000913f5692e59ea005-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.414 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.415 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.415 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.415 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.415 DEBUG:boto:Method: GET 2023-09-29T22:22:49.415 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.415 DEBUG:boto:Data: 2023-09-29T22:22:49.415 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.415 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.415 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.415 DEBUG:boto:Params: {} 2023-09-29T22:22:49.416 DEBUG:boto:Token: None 2023-09-29T22:22:49.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a849fb73811183b2-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.419 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.419 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.419 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.419 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.419 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.419 DEBUG:boto:Method: GET 2023-09-29T22:22:49.419 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.419 DEBUG:boto:Data: 2023-09-29T22:22:49.420 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.420 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.420 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.420 DEBUG:boto:Params: {} 2023-09-29T22:22:49.420 DEBUG:boto:Token: None 2023-09-29T22:22:49.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccc06a4ba63cdaa-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.423 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.423 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.423 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.424 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.424 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.424 DEBUG:boto:Data: 2023-09-29T22:22:49.424 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.424 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.424 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.424 DEBUG:boto:Params: {} 2023-09-29T22:22:49.424 DEBUG:boto:Token: None 2023-09-29T22:22:49.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000475d22a2c9a67a44-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.427 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.427 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.427 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.428 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.428 DEBUG:boto:Data: 2023-09-29T22:22:49.428 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.428 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.428 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.428 DEBUG:boto:Params: {} 2023-09-29T22:22:49.428 DEBUG:boto:Token: None 2023-09-29T22:22:49.428 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000817f0ed2f35250b7-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.431 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.431 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.431 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.431 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.432 DEBUG:boto:Method: GET 2023-09-29T22:22:49.432 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.432 DEBUG:boto:Data: 2023-09-29T22:22:49.432 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.432 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.432 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.432 DEBUG:boto:Params: {} 2023-09-29T22:22:49.432 DEBUG:boto:Token: None 2023-09-29T22:22:49.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.433 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d452b96b34cf05da-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.478 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.478 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.479 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.479 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.479 DEBUG:boto:Method: GET 2023-09-29T22:22:49.479 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.479 DEBUG:boto:Data: 2023-09-29T22:22:49.479 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.479 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.479 DEBUG:boto:Params: {} 2023-09-29T22:22:49.479 DEBUG:boto:Token: None 2023-09-29T22:22:49.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ab8b128bc0c9481-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.481 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.481 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.482 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.482 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.482 DEBUG:boto:Method: GET 2023-09-29T22:22:49.482 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.482 DEBUG:boto:Data: 2023-09-29T22:22:49.482 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.482 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.482 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.482 DEBUG:boto:Params: {} 2023-09-29T22:22:49.482 DEBUG:boto:Token: None 2023-09-29T22:22:49.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606c86c968597c3f-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.485 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.486 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.486 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.486 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.486 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.486 DEBUG:boto:Method: GET 2023-09-29T22:22:49.486 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.486 DEBUG:boto:Data: 2023-09-29T22:22:49.486 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.486 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.486 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.487 DEBUG:boto:Params: {} 2023-09-29T22:22:49.487 DEBUG:boto:Token: None 2023-09-29T22:22:49.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.487 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029262185682dae98-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.489 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.490 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.490 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.490 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.490 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.490 DEBUG:boto:Method: GET 2023-09-29T22:22:49.490 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.490 DEBUG:boto:Data: 2023-09-29T22:22:49.490 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.490 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.491 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.491 DEBUG:boto:Params: {} 2023-09-29T22:22:49.491 DEBUG:boto:Token: None 2023-09-29T22:22:49.491 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g= 2023-09-29T22:22:49.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g=', 'Content-Length': '0'} 2023-09-29T22:22:49.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053d6922288cf4210-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.493 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.493 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.493 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.493 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.493 DEBUG:boto:Method: GET 2023-09-29T22:22:49.493 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.493 DEBUG:boto:Data: 2023-09-29T22:22:49.493 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.494 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.494 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.494 DEBUG:boto:Params: {} 2023-09-29T22:22:49.494 DEBUG:boto:Token: None 2023-09-29T22:22:49.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.494 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g= 2023-09-29T22:22:49.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g=', 'Content-Length': '0'} 2023-09-29T22:22:49.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001709777700dec6ad-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.538 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.538 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.538 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.539 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.539 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.539 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.539 DEBUG:boto:Data: 2023-09-29T22:22:49.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.539 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.539 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.539 DEBUG:boto:Params: {} 2023-09-29T22:22:49.539 DEBUG:boto:Token: None 2023-09-29T22:22:49.539 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U34Xmgc0S/SVOZ2b4fmuRd0Ax9Y= 2023-09-29T22:22:49.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U34Xmgc0S/SVOZ2b4fmuRd0Ax9Y=', 'Content-Length': '0'} 2023-09-29T22:22:49.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8fecde8dd088b4c-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.541 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.541 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.542 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.542 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.542 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.542 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.542 DEBUG:boto:Data: 2023-09-29T22:22:49.542 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.542 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.542 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.542 DEBUG:boto:Params: {} 2023-09-29T22:22:49.542 DEBUG:boto:Token: None 2023-09-29T22:22:49.543 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.543 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U34Xmgc0S/SVOZ2b4fmuRd0Ax9Y= 2023-09-29T22:22:49.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U34Xmgc0S/SVOZ2b4fmuRd0Ax9Y=', 'Content-Length': '0'} 2023-09-29T22:22:49.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcf40c2240c49c9a-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.545 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.545 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.545 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.545 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.545 DEBUG:boto:Method: GET 2023-09-29T22:22:49.545 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.545 DEBUG:boto:Data: 2023-09-29T22:22:49.545 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.545 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.545 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.545 DEBUG:boto:Params: {} 2023-09-29T22:22:49.546 DEBUG:boto:Token: None 2023-09-29T22:22:49.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.546 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g= 2023-09-29T22:22:49.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g=', 'Content-Length': '0'} 2023-09-29T22:22:49.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008074c3d0e1f8a84-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.590 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.590 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.591 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.591 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.591 DEBUG:boto:Method: GET 2023-09-29T22:22:49.591 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.591 DEBUG:boto:Data: 2023-09-29T22:22:49.591 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.591 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.591 DEBUG:boto:Params: {} 2023-09-29T22:22:49.591 DEBUG:boto:Token: None 2023-09-29T22:22:49.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:49.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g= 2023-09-29T22:22:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b/NcqBehgCnk4+TtkUa5RZpEg/g=', 'Content-Length': '0'} 2023-09-29T22:22:49.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0b02995983637ba-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.634 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.634 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.635 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.635 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.635 DEBUG:boto:Method: GET 2023-09-29T22:22:49.635 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.635 DEBUG:boto:Data: 2023-09-29T22:22:49.635 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.635 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.635 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.635 DEBUG:boto:Params: {} 2023-09-29T22:22:49.635 DEBUG:boto:Token: None 2023-09-29T22:22:49.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03ef13327a29a4e-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.638 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.639 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.639 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.639 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.639 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.639 DEBUG:boto:Method: GET 2023-09-29T22:22:49.639 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.639 DEBUG:boto:Data: 2023-09-29T22:22:49.639 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.639 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.639 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.640 DEBUG:boto:Params: {} 2023-09-29T22:22:49.640 DEBUG:boto:Token: None 2023-09-29T22:22:49.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d885f716a42da03c-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.642 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.643 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.643 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.643 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.643 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.643 DEBUG:boto:Data: 2023-09-29T22:22:49.643 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.643 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.643 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.643 DEBUG:boto:Params: {} 2023-09-29T22:22:49.644 DEBUG:boto:Token: None 2023-09-29T22:22:49.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.644 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4bb74c92af563ac-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.647 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.647 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.648 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.648 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.648 DEBUG:boto:Data: 2023-09-29T22:22:49.648 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.648 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.648 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.648 DEBUG:boto:Params: {} 2023-09-29T22:22:49.648 DEBUG:boto:Token: None 2023-09-29T22:22:49.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.649 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000652bc113a2747a27-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.652 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.652 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.652 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.652 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.652 DEBUG:boto:Method: GET 2023-09-29T22:22:49.652 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.652 DEBUG:boto:Data: 2023-09-29T22:22:49.652 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.652 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.652 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.653 DEBUG:boto:Params: {} 2023-09-29T22:22:49.653 DEBUG:boto:Token: None 2023-09-29T22:22:49.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.653 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d09501ee8b2ce75-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.698 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.698 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.698 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.699 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.699 DEBUG:boto:Method: GET 2023-09-29T22:22:49.699 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.699 DEBUG:boto:Data: 2023-09-29T22:22:49.699 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.699 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.699 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.699 DEBUG:boto:Params: {} 2023-09-29T22:22:49.699 DEBUG:boto:Token: None 2023-09-29T22:22:49.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d4abdb536af16a9-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.702 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.702 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.702 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.702 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.703 DEBUG:boto:Method: GET 2023-09-29T22:22:49.703 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.703 DEBUG:boto:Data: 2023-09-29T22:22:49.703 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.703 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.703 DEBUG:boto:Params: {} 2023-09-29T22:22:49.703 DEBUG:boto:Token: None 2023-09-29T22:22:49.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.703 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ed2dc95cfdb4fd-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.706 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.706 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.706 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.706 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.707 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.707 DEBUG:boto:Method: GET 2023-09-29T22:22:49.707 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.707 DEBUG:boto:Data: 2023-09-29T22:22:49.707 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.707 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.707 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.707 DEBUG:boto:Params: {} 2023-09-29T22:22:49.707 DEBUG:boto:Token: None 2023-09-29T22:22:49.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.708 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d53af2f6d02bf8-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.710 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.711 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.711 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.711 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.711 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.711 DEBUG:boto:Method: GET 2023-09-29T22:22:49.711 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.711 DEBUG:boto:Data: 2023-09-29T22:22:49.711 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.712 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.712 DEBUG:boto:Params: {} 2023-09-29T22:22:49.712 DEBUG:boto:Token: None 2023-09-29T22:22:49.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g= 2023-09-29T22:22:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g=', 'Content-Length': '0'} 2023-09-29T22:22:49.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096143046368c6d5d-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.714 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.714 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.714 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.714 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.714 DEBUG:boto:Method: GET 2023-09-29T22:22:49.714 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.715 DEBUG:boto:Data: 2023-09-29T22:22:49.715 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.715 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.715 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.715 DEBUG:boto:Params: {} 2023-09-29T22:22:49.715 DEBUG:boto:Token: None 2023-09-29T22:22:49.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.715 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g= 2023-09-29T22:22:49.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g=', 'Content-Length': '0'} 2023-09-29T22:22:49.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdb69c6f68d9aa3a-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.717 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.717 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.718 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.718 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.718 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.718 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.718 DEBUG:boto:Data: 2023-09-29T22:22:49.718 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.718 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.718 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.718 DEBUG:boto:Params: {} 2023-09-29T22:22:49.718 DEBUG:boto:Token: None 2023-09-29T22:22:49.719 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eZxCxwGjzKkdDW3xkKwOEm6xKls= 2023-09-29T22:22:49.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eZxCxwGjzKkdDW3xkKwOEm6xKls=', 'Content-Length': '0'} 2023-09-29T22:22:49.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d359a5b3f7d8bb0-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.721 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.721 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.721 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.721 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.721 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.721 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.721 DEBUG:boto:Data: 2023-09-29T22:22:49.721 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.721 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.721 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.721 DEBUG:boto:Params: {} 2023-09-29T22:22:49.722 DEBUG:boto:Token: None 2023-09-29T22:22:49.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.722 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eZxCxwGjzKkdDW3xkKwOEm6xKls= 2023-09-29T22:22:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eZxCxwGjzKkdDW3xkKwOEm6xKls=', 'Content-Length': '0'} 2023-09-29T22:22:49.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa09610a93623f24-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.724 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.724 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.724 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.724 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.724 DEBUG:boto:Method: GET 2023-09-29T22:22:49.725 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.725 DEBUG:boto:Data: 2023-09-29T22:22:49.725 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.725 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.725 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.725 DEBUG:boto:Params: {} 2023-09-29T22:22:49.725 DEBUG:boto:Token: None 2023-09-29T22:22:49.725 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.725 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g= 2023-09-29T22:22:49.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g=', 'Content-Length': '0'} 2023-09-29T22:22:49.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef46431c1ec56eb-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.727 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.727 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.727 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.727 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.727 DEBUG:boto:Method: GET 2023-09-29T22:22:49.728 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.728 DEBUG:boto:Data: 2023-09-29T22:22:49.728 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.728 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.728 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.728 DEBUG:boto:Params: {} 2023-09-29T22:22:49.728 DEBUG:boto:Token: None 2023-09-29T22:22:49.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:49.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g= 2023-09-29T22:22:49.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7YYb/4xSWUVQlRpa8SWTHNcfR8g=', 'Content-Length': '0'} 2023-09-29T22:22:49.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000595ba787bb3a6791-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.770 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.770 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.771 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.771 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.771 DEBUG:boto:Method: GET 2023-09-29T22:22:49.771 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.771 DEBUG:boto:Data: 2023-09-29T22:22:49.771 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.771 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.771 DEBUG:boto:Params: {} 2023-09-29T22:22:49.771 DEBUG:boto:Token: None 2023-09-29T22:22:49.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9fc062b9a3cadf-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.774 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.775 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.775 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.775 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.775 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.775 DEBUG:boto:Method: GET 2023-09-29T22:22:49.775 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.775 DEBUG:boto:Data: 2023-09-29T22:22:49.775 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.775 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.775 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.775 DEBUG:boto:Params: {} 2023-09-29T22:22:49.776 DEBUG:boto:Token: None 2023-09-29T22:22:49.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.776 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a0e0112129ade5a-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.778 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.779 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.779 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.779 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.779 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.779 DEBUG:boto:Data: 2023-09-29T22:22:49.779 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.779 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.779 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.780 DEBUG:boto:Params: {} 2023-09-29T22:22:49.780 DEBUG:boto:Token: None 2023-09-29T22:22:49.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003141e883f70977fa-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.783 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.783 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.783 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.783 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.783 DEBUG:boto:Data: 2023-09-29T22:22:49.783 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.783 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.783 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.784 DEBUG:boto:Params: {} 2023-09-29T22:22:49.784 DEBUG:boto:Token: None 2023-09-29T22:22:49.784 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.784 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c747eb5be83d797e-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.787 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.787 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.787 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.787 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.787 DEBUG:boto:Method: GET 2023-09-29T22:22:49.787 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.788 DEBUG:boto:Data: 2023-09-29T22:22:49.788 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.788 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.788 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.788 DEBUG:boto:Params: {} 2023-09-29T22:22:49.788 DEBUG:boto:Token: None 2023-09-29T22:22:49.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baa89c043c963374-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.830 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.830 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.831 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.831 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.831 DEBUG:boto:Method: GET 2023-09-29T22:22:49.831 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.831 DEBUG:boto:Data: 2023-09-29T22:22:49.831 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.831 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.831 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.831 DEBUG:boto:Params: {} 2023-09-29T22:22:49.832 DEBUG:boto:Token: None 2023-09-29T22:22:49.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077aaf5df133df273-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.834 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.834 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.834 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.834 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.834 DEBUG:boto:Method: GET 2023-09-29T22:22:49.834 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.835 DEBUG:boto:Data: 2023-09-29T22:22:49.835 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.835 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.835 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.835 DEBUG:boto:Params: {} 2023-09-29T22:22:49.835 DEBUG:boto:Token: None 2023-09-29T22:22:49.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ebc21fde7c897a9-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.838 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.838 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.838 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.839 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.839 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.839 DEBUG:boto:Method: GET 2023-09-29T22:22:49.839 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.839 DEBUG:boto:Data: 2023-09-29T22:22:49.839 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.839 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.839 DEBUG:boto:Params: {} 2023-09-29T22:22:49.839 DEBUG:boto:Token: None 2023-09-29T22:22:49.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000925697821c91b339-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.842 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.843 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.843 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.843 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.843 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.843 DEBUG:boto:Method: GET 2023-09-29T22:22:49.843 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.843 DEBUG:boto:Data: 2023-09-29T22:22:49.843 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.843 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.843 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.844 DEBUG:boto:Params: {} 2023-09-29T22:22:49.844 DEBUG:boto:Token: None 2023-09-29T22:22:49.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.844 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc= 2023-09-29T22:22:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc=', 'Content-Length': '0'} 2023-09-29T22:22:49.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcdbd74716b9a6ac-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.846 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.846 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.846 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.846 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.846 DEBUG:boto:Method: GET 2023-09-29T22:22:49.846 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.847 DEBUG:boto:Data: 2023-09-29T22:22:49.847 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.847 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.847 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.847 DEBUG:boto:Params: {} 2023-09-29T22:22:49.847 DEBUG:boto:Token: None 2023-09-29T22:22:49.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc= 2023-09-29T22:22:49.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc=', 'Content-Length': '0'} 2023-09-29T22:22:49.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a253219076371750-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.849 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.849 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.849 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.850 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.850 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.850 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.850 DEBUG:boto:Data: 2023-09-29T22:22:49.850 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.850 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.850 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.850 DEBUG:boto:Params: {} 2023-09-29T22:22:49.850 DEBUG:boto:Token: None 2023-09-29T22:22:49.850 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.851 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:poYn8Q36m12iGAxDwRMoLHfiVVc= 2023-09-29T22:22:49.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:poYn8Q36m12iGAxDwRMoLHfiVVc=', 'Content-Length': '0'} 2023-09-29T22:22:49.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c632c64a46288b2d-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.853 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.853 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.853 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.853 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.853 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.853 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.853 DEBUG:boto:Data: 2023-09-29T22:22:49.853 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.853 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.853 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.853 DEBUG:boto:Params: {} 2023-09-29T22:22:49.854 DEBUG:boto:Token: None 2023-09-29T22:22:49.854 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.854 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:poYn8Q36m12iGAxDwRMoLHfiVVc= 2023-09-29T22:22:49.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:poYn8Q36m12iGAxDwRMoLHfiVVc=', 'Content-Length': '0'} 2023-09-29T22:22:49.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b08b2a37bed79e77-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.856 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.856 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.856 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.856 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.856 DEBUG:boto:Method: GET 2023-09-29T22:22:49.856 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.856 DEBUG:boto:Data: 2023-09-29T22:22:49.857 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.857 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.857 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.857 DEBUG:boto:Params: {} 2023-09-29T22:22:49.857 DEBUG:boto:Token: None 2023-09-29T22:22:49.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.857 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc= 2023-09-29T22:22:49.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc=', 'Content-Length': '0'} 2023-09-29T22:22:49.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afceb59818b00775-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.902 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.902 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.903 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.903 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.903 DEBUG:boto:Method: GET 2023-09-29T22:22:49.903 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.903 DEBUG:boto:Data: 2023-09-29T22:22:49.903 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.903 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.903 DEBUG:boto:Params: {} 2023-09-29T22:22:49.904 DEBUG:boto:Token: None 2023-09-29T22:22:49.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:49.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc= 2023-09-29T22:22:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hw90A67WY//yy9p17S7Uejy8nwc=', 'Content-Length': '0'} 2023-09-29T22:22:49.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f3fb16380308925-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.906 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.906 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.906 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.906 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.906 DEBUG:boto:Method: GET 2023-09-29T22:22:49.906 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.907 DEBUG:boto:Data: 2023-09-29T22:22:49.907 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.907 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.907 DEBUG:boto:Params: {} 2023-09-29T22:22:49.907 DEBUG:boto:Token: None 2023-09-29T22:22:49.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.907 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3fcf96cfae0101-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.910 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.910 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.910 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.910 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.911 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.911 DEBUG:boto:Method: GET 2023-09-29T22:22:49.911 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.911 DEBUG:boto:Data: 2023-09-29T22:22:49.911 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.911 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.911 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.911 DEBUG:boto:Params: {} 2023-09-29T22:22:49.911 DEBUG:boto:Token: None 2023-09-29T22:22:49.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.912 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786f959e69d31f18-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.914 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.914 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.915 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.915 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.915 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.915 DEBUG:boto:Data: 2023-09-29T22:22:49.915 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.915 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.915 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.915 DEBUG:boto:Params: {} 2023-09-29T22:22:49.915 DEBUG:boto:Token: None 2023-09-29T22:22:49.916 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.916 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009b07bb81c5336c6-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.919 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.919 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.919 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.919 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:49.919 DEBUG:boto:Data: 2023-09-29T22:22:49.919 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.919 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.919 DEBUG:boto:Params: {} 2023-09-29T22:22:49.920 DEBUG:boto:Token: None 2023-09-29T22:22:49.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj 2023-09-29T22:22:49.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0= 2023-09-29T22:22:49.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ACj72U7+i54sA8wKdup6TSmaKh0=', 'Content-Length': '0'} 2023-09-29T22:22:49.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2ecc59f81951709-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.923 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.923 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.923 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.923 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.924 DEBUG:boto:Method: GET 2023-09-29T22:22:49.924 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.924 DEBUG:boto:Data: 2023-09-29T22:22:49.924 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.924 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.924 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.924 DEBUG:boto:Params: {} 2023-09-29T22:22:49.924 DEBUG:boto:Token: None 2023-09-29T22:22:49.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d30535bfaa05f0cb-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.966 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.966 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.966 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.967 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.967 DEBUG:boto:Method: GET 2023-09-29T22:22:49.967 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.967 DEBUG:boto:Data: 2023-09-29T22:22:49.967 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.967 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.967 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.967 DEBUG:boto:Params: {} 2023-09-29T22:22:49.967 DEBUG:boto:Token: None 2023-09-29T22:22:49.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:49.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc= 2023-09-29T22:22:49.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5eKMCEjDQXuugmNMYstqBJtbQgc=', 'Content-Length': '0'} 2023-09-29T22:22:49.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068dc5f3f14727903-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.970 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.970 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.970 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.970 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.970 DEBUG:boto:Method: GET 2023-09-29T22:22:49.970 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.971 DEBUG:boto:Data: 2023-09-29T22:22:49.971 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.971 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.971 DEBUG:boto:Params: {} 2023-09-29T22:22:49.971 DEBUG:boto:Token: None 2023-09-29T22:22:49.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.971 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b950621fb38f2d5-0065174e39-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.974 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.975 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:49.975 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:49.975 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.975 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.975 DEBUG:boto:Method: GET 2023-09-29T22:22:49.975 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:49.975 DEBUG:boto:Data: 2023-09-29T22:22:49.975 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.976 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.976 DEBUG:boto:Params: {} 2023-09-29T22:22:49.976 DEBUG:boto:Token: None 2023-09-29T22:22:49.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/ 2023-09-29T22:22:49.976 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8= 2023-09-29T22:22:49.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kri5Zv+p7ZK5wQmMxzArtUCsFi8=', 'Content-Length': '0'} 2023-09-29T22:22:49.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5dbf7e201720b3d-0065174e39-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.978 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:49.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.979 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.979 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.979 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.979 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.979 DEBUG:boto:Method: GET 2023-09-29T22:22:49.979 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.979 DEBUG:boto:Data: 2023-09-29T22:22:49.980 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.980 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.980 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.980 DEBUG:boto:Params: {} 2023-09-29T22:22:49.980 DEBUG:boto:Token: None 2023-09-29T22:22:49.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.980 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow= 2023-09-29T22:22:49.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow=', 'Content-Length': '0'} 2023-09-29T22:22:49.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e78c51e9bc838de-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.982 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.982 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.983 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.983 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.983 DEBUG:boto:Method: GET 2023-09-29T22:22:49.983 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.983 DEBUG:boto:Data: 2023-09-29T22:22:49.983 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.983 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.983 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.983 DEBUG:boto:Params: {} 2023-09-29T22:22:49.983 DEBUG:boto:Token: None 2023-09-29T22:22:49.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.984 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow= 2023-09-29T22:22:49.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow=', 'Content-Length': '0'} 2023-09-29T22:22:49.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce5ea42c95619e88-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.986 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.986 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.986 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.986 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.986 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.986 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.986 DEBUG:boto:Data: 2023-09-29T22:22:49.986 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.986 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.987 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.987 DEBUG:boto:Params: {} 2023-09-29T22:22:49.987 DEBUG:boto:Token: None 2023-09-29T22:22:49.987 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.987 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DbscMqDF9fez2lTLa7pS4KpxKLA= 2023-09-29T22:22:49.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DbscMqDF9fez2lTLa7pS4KpxKLA=', 'Content-Length': '0'} 2023-09-29T22:22:49.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3c38dcea81b027b-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.989 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.989 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.989 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.989 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.989 DEBUG:boto:Method: HEAD 2023-09-29T22:22:49.989 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.989 DEBUG:boto:Data: 2023-09-29T22:22:49.990 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.990 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:49.990 DEBUG:boto:Port: 8000 2023-09-29T22:22:49.990 DEBUG:boto:Params: {} 2023-09-29T22:22:49.990 DEBUG:boto:Token: None 2023-09-29T22:22:49.990 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.990 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DbscMqDF9fez2lTLa7pS4KpxKLA= 2023-09-29T22:22:49.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DbscMqDF9fez2lTLa7pS4KpxKLA=', 'Content-Length': '0'} 2023-09-29T22:22:49.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d309e34d1054719-0065174e39-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:49.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:49.992 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:49.992 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:49.992 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.993 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.993 DEBUG:boto:Method: GET 2023-09-29T22:22:49.993 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.993 DEBUG:boto:Data: 2023-09-29T22:22:49.993 DEBUG:boto:Headers: {} 2023-09-29T22:22:49.993 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:49.993 DEBUG:boto:Port: 8001 2023-09-29T22:22:49.993 DEBUG:boto:Params: {} 2023-09-29T22:22:49.993 DEBUG:boto:Token: None 2023-09-29T22:22:49.993 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:49 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:49.994 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow= 2023-09-29T22:22:49.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MTSfME4kfG+6dl8ZtJkvwFkfFow=', 'Content-Length': '0'} 2023-09-29T22:22:49.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d43cf21a0984b3b4-0065174e39-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.038 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.038 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.039 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:50.039 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:50.039 DEBUG:boto:Method: GET 2023-09-29T22:22:50.039 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:50.039 DEBUG:boto:Data: 2023-09-29T22:22:50.039 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.039 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.039 DEBUG:boto:Params: {} 2023-09-29T22:22:50.039 DEBUG:boto:Token: None 2023-09-29T22:22:50.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:50.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0jM/Oi1s0ixN8QGRaAzzpXkdziE= 2023-09-29T22:22:50.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0jM/Oi1s0ixN8QGRaAzzpXkdziE=', 'Content-Length': '0'} 2023-09-29T22:22:50.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001facc50e90c8057a-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.082 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.082 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.083 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.083 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.083 DEBUG:boto:Method: GET 2023-09-29T22:22:50.083 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.083 DEBUG:boto:Data: 2023-09-29T22:22:50.083 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.083 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.083 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.083 DEBUG:boto:Params: {} 2023-09-29T22:22:50.083 DEBUG:boto:Token: None 2023-09-29T22:22:50.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8dd7e6ea7bcd86f-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.087 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.087 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.087 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.087 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.087 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.088 DEBUG:boto:Method: GET 2023-09-29T22:22:50.088 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.088 DEBUG:boto:Data: 2023-09-29T22:22:50.088 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.088 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.088 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.088 DEBUG:boto:Params: {} 2023-09-29T22:22:50.088 DEBUG:boto:Token: None 2023-09-29T22:22:50.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e679d514f769c15-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.134 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.135 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.135 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.135 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.135 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.135 DEBUG:boto:Data: 2023-09-29T22:22:50.135 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.135 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.136 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.136 DEBUG:boto:Params: {} 2023-09-29T22:22:50.136 DEBUG:boto:Token: None 2023-09-29T22:22:50.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.136 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013bf9bb1c20f6d5c-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.140 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.140 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.140 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.140 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.140 DEBUG:boto:Data: 2023-09-29T22:22:50.140 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.140 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.140 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.140 DEBUG:boto:Params: {} 2023-09-29T22:22:50.141 DEBUG:boto:Token: None 2023-09-29T22:22:50.141 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f7b9fc75c7fa81c-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.144 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.145 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.145 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.145 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.145 DEBUG:boto:Method: GET 2023-09-29T22:22:50.145 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.145 DEBUG:boto:Data: 2023-09-29T22:22:50.145 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.145 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.145 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.145 DEBUG:boto:Params: {} 2023-09-29T22:22:50.146 DEBUG:boto:Token: None 2023-09-29T22:22:50.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.146 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c33c2625b6eb9016-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.190 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.191 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.191 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.191 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.191 DEBUG:boto:Method: GET 2023-09-29T22:22:50.191 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.191 DEBUG:boto:Data: 2023-09-29T22:22:50.191 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.191 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.191 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.191 DEBUG:boto:Params: {} 2023-09-29T22:22:50.192 DEBUG:boto:Token: None 2023-09-29T22:22:50.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.192 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac806dc9955096e0-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.194 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.194 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.194 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.195 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.195 DEBUG:boto:Method: GET 2023-09-29T22:22:50.195 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.195 DEBUG:boto:Data: 2023-09-29T22:22:50.195 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.195 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.195 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.195 DEBUG:boto:Params: {} 2023-09-29T22:22:50.195 DEBUG:boto:Token: None 2023-09-29T22:22:50.195 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.196 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007088e47c147f3b84-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.242 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.243 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.243 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.243 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.243 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.243 DEBUG:boto:Method: GET 2023-09-29T22:22:50.243 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.243 DEBUG:boto:Data: 2023-09-29T22:22:50.244 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.244 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.244 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.244 DEBUG:boto:Params: {} 2023-09-29T22:22:50.244 DEBUG:boto:Token: None 2023-09-29T22:22:50.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038113de107a9d6a-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.247 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.247 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.248 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.248 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.248 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.248 DEBUG:boto:Method: GET 2023-09-29T22:22:50.248 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.248 DEBUG:boto:Data: 2023-09-29T22:22:50.248 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.248 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.248 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.248 DEBUG:boto:Params: {} 2023-09-29T22:22:50.249 DEBUG:boto:Token: None 2023-09-29T22:22:50.249 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.249 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY= 2023-09-29T22:22:50.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY=', 'Content-Length': '0'} 2023-09-29T22:22:50.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c09001755a6bdc2-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.251 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.251 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.251 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.251 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.251 DEBUG:boto:Method: GET 2023-09-29T22:22:50.252 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.252 DEBUG:boto:Data: 2023-09-29T22:22:50.252 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.252 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.252 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.252 DEBUG:boto:Params: {} 2023-09-29T22:22:50.252 DEBUG:boto:Token: None 2023-09-29T22:22:50.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY= 2023-09-29T22:22:50.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY=', 'Content-Length': '0'} 2023-09-29T22:22:50.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003adc4730bc1b66d6-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.255 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.255 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.255 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.255 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.255 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.255 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.255 DEBUG:boto:Data: 2023-09-29T22:22:50.255 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.255 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.255 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.255 DEBUG:boto:Params: {} 2023-09-29T22:22:50.256 DEBUG:boto:Token: None 2023-09-29T22:22:50.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m4DQu8PCs+XrpaApGQCnQu/o9ow= 2023-09-29T22:22:50.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m4DQu8PCs+XrpaApGQCnQu/o9ow=', 'Content-Length': '0'} 2023-09-29T22:22:50.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000375da98c6ed6a433-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.258 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.258 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.258 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.258 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.258 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.258 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.258 DEBUG:boto:Data: 2023-09-29T22:22:50.258 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.258 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.259 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.259 DEBUG:boto:Params: {} 2023-09-29T22:22:50.259 DEBUG:boto:Token: None 2023-09-29T22:22:50.259 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.259 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m4DQu8PCs+XrpaApGQCnQu/o9ow= 2023-09-29T22:22:50.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m4DQu8PCs+XrpaApGQCnQu/o9ow=', 'Content-Length': '0'} 2023-09-29T22:22:50.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7fe837d0cde56c3-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.261 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.261 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.261 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.261 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.261 DEBUG:boto:Method: GET 2023-09-29T22:22:50.261 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.262 DEBUG:boto:Data: 2023-09-29T22:22:50.262 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.262 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.262 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.262 DEBUG:boto:Params: {} 2023-09-29T22:22:50.262 DEBUG:boto:Token: None 2023-09-29T22:22:50.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.262 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY= 2023-09-29T22:22:50.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY=', 'Content-Length': '0'} 2023-09-29T22:22:50.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048958a9001230eac-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.264 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.265 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.265 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.265 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.265 DEBUG:boto:Method: GET 2023-09-29T22:22:50.265 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.265 DEBUG:boto:Data: 2023-09-29T22:22:50.265 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.265 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.265 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.265 DEBUG:boto:Params: {} 2023-09-29T22:22:50.266 DEBUG:boto:Token: None 2023-09-29T22:22:50.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:50.266 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY= 2023-09-29T22:22:50.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/eyco+mlk6POd95+9oZYsPziDqY=', 'Content-Length': '0'} 2023-09-29T22:22:50.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000171baeb85869442f-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.310 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.310 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.310 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.311 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.311 DEBUG:boto:Method: GET 2023-09-29T22:22:50.311 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.311 DEBUG:boto:Data: 2023-09-29T22:22:50.311 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.311 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.311 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.311 DEBUG:boto:Params: {} 2023-09-29T22:22:50.311 DEBUG:boto:Token: None 2023-09-29T22:22:50.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3cf4c670e3b5d6-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.314 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.315 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.315 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.315 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.315 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.315 DEBUG:boto:Method: GET 2023-09-29T22:22:50.315 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.315 DEBUG:boto:Data: 2023-09-29T22:22:50.315 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.316 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.316 DEBUG:boto:Params: {} 2023-09-29T22:22:50.316 DEBUG:boto:Token: None 2023-09-29T22:22:50.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f4b248ddfaad65-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.319 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.319 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.319 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.319 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.319 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.319 DEBUG:boto:Data: 2023-09-29T22:22:50.319 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.319 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.320 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.320 DEBUG:boto:Params: {} 2023-09-29T22:22:50.320 DEBUG:boto:Token: None 2023-09-29T22:22:50.320 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.320 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f91706ec4936d780-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.323 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.323 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.324 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.324 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.324 DEBUG:boto:Data: 2023-09-29T22:22:50.324 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.324 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.324 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.324 DEBUG:boto:Params: {} 2023-09-29T22:22:50.324 DEBUG:boto:Token: None 2023-09-29T22:22:50.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ac8246e887f6026-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.328 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.328 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.328 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.328 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.328 DEBUG:boto:Method: GET 2023-09-29T22:22:50.328 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.328 DEBUG:boto:Data: 2023-09-29T22:22:50.328 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.328 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.329 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.329 DEBUG:boto:Params: {} 2023-09-29T22:22:50.329 DEBUG:boto:Token: None 2023-09-29T22:22:50.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.329 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040df01e25e6e7a76-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.374 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.374 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.374 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.374 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.375 DEBUG:boto:Method: GET 2023-09-29T22:22:50.375 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.375 DEBUG:boto:Data: 2023-09-29T22:22:50.375 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.375 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.375 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.375 DEBUG:boto:Params: {} 2023-09-29T22:22:50.375 DEBUG:boto:Token: None 2023-09-29T22:22:50.375 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.376 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003063d1be912fd17f-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.378 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.378 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.378 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.378 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.378 DEBUG:boto:Method: GET 2023-09-29T22:22:50.378 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.378 DEBUG:boto:Data: 2023-09-29T22:22:50.378 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.378 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.379 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.379 DEBUG:boto:Params: {} 2023-09-29T22:22:50.379 DEBUG:boto:Token: None 2023-09-29T22:22:50.379 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.379 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006055721a1217f492-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.382 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.382 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.382 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.382 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.382 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.383 DEBUG:boto:Method: GET 2023-09-29T22:22:50.383 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.383 DEBUG:boto:Data: 2023-09-29T22:22:50.383 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.383 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.383 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.383 DEBUG:boto:Params: {} 2023-09-29T22:22:50.383 DEBUG:boto:Token: None 2023-09-29T22:22:50.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.383 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953a79848f185269-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.386 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.386 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.386 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.387 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.387 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.387 DEBUG:boto:Method: GET 2023-09-29T22:22:50.387 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.387 DEBUG:boto:Data: 2023-09-29T22:22:50.387 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.387 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.387 DEBUG:boto:Params: {} 2023-09-29T22:22:50.387 DEBUG:boto:Token: None 2023-09-29T22:22:50.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU= 2023-09-29T22:22:50.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU=', 'Content-Length': '0'} 2023-09-29T22:22:50.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bacf630e1fa8aca-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.390 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.390 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.390 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.390 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.390 DEBUG:boto:Method: GET 2023-09-29T22:22:50.390 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.390 DEBUG:boto:Data: 2023-09-29T22:22:50.391 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.391 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.391 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.391 DEBUG:boto:Params: {} 2023-09-29T22:22:50.391 DEBUG:boto:Token: None 2023-09-29T22:22:50.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.391 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU= 2023-09-29T22:22:50.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU=', 'Content-Length': '0'} 2023-09-29T22:22:50.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d929f5373eb1ecd-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.393 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.394 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.394 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.394 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.394 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.394 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.394 DEBUG:boto:Data: 2023-09-29T22:22:50.394 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.394 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.394 DEBUG:boto:Params: {} 2023-09-29T22:22:50.395 DEBUG:boto:Token: None 2023-09-29T22:22:50.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qqSetjcF2yXR2ruKf/GWeFmLoH0= 2023-09-29T22:22:50.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qqSetjcF2yXR2ruKf/GWeFmLoH0=', 'Content-Length': '0'} 2023-09-29T22:22:50.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c1a05939060c7ab-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.397 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.397 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.397 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.397 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.397 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.397 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.397 DEBUG:boto:Data: 2023-09-29T22:22:50.397 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.397 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.397 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.397 DEBUG:boto:Params: {} 2023-09-29T22:22:50.398 DEBUG:boto:Token: None 2023-09-29T22:22:50.398 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.398 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qqSetjcF2yXR2ruKf/GWeFmLoH0= 2023-09-29T22:22:50.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qqSetjcF2yXR2ruKf/GWeFmLoH0=', 'Content-Length': '0'} 2023-09-29T22:22:50.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dc837beb0674ad2-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.400 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.400 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.400 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.400 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.401 DEBUG:boto:Method: GET 2023-09-29T22:22:50.401 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.401 DEBUG:boto:Data: 2023-09-29T22:22:50.401 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.401 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.401 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.401 DEBUG:boto:Params: {} 2023-09-29T22:22:50.401 DEBUG:boto:Token: None 2023-09-29T22:22:50.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.401 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU= 2023-09-29T22:22:50.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU=', 'Content-Length': '0'} 2023-09-29T22:22:50.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9e5d32b3780c6e3-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.446 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.447 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.447 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.447 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.447 DEBUG:boto:Method: GET 2023-09-29T22:22:50.447 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.447 DEBUG:boto:Data: 2023-09-29T22:22:50.447 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.447 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.447 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.447 DEBUG:boto:Params: {} 2023-09-29T22:22:50.448 DEBUG:boto:Token: None 2023-09-29T22:22:50.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:50.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU= 2023-09-29T22:22:50.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nwp+L5Hb1rmYj1ifCNBLrvRlyiU=', 'Content-Length': '0'} 2023-09-29T22:22:50.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eaa442d9e1cb768-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.450 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.450 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.450 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.450 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.451 DEBUG:boto:Method: GET 2023-09-29T22:22:50.451 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.451 DEBUG:boto:Data: 2023-09-29T22:22:50.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.451 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.451 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.451 DEBUG:boto:Params: {} 2023-09-29T22:22:50.451 DEBUG:boto:Token: None 2023-09-29T22:22:50.451 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b184f3dc593ff5d-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.454 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.455 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.455 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.455 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.455 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.455 DEBUG:boto:Method: GET 2023-09-29T22:22:50.455 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.455 DEBUG:boto:Data: 2023-09-29T22:22:50.455 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.456 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.456 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.456 DEBUG:boto:Params: {} 2023-09-29T22:22:50.456 DEBUG:boto:Token: None 2023-09-29T22:22:50.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3e1bc28e28cd916-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.459 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.459 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.459 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.459 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.460 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.460 DEBUG:boto:Data: 2023-09-29T22:22:50.460 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.460 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.460 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.460 DEBUG:boto:Params: {} 2023-09-29T22:22:50.460 DEBUG:boto:Token: None 2023-09-29T22:22:50.460 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eced0d1fca4966c-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.464 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.464 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.464 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.464 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.464 DEBUG:boto:Data: 2023-09-29T22:22:50.464 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.464 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.464 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.465 DEBUG:boto:Params: {} 2023-09-29T22:22:50.465 DEBUG:boto:Token: None 2023-09-29T22:22:50.465 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.465 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8f45dbb229d27ab-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.468 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.468 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.468 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.468 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.468 DEBUG:boto:Method: GET 2023-09-29T22:22:50.469 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.469 DEBUG:boto:Data: 2023-09-29T22:22:50.469 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.469 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.469 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.469 DEBUG:boto:Params: {} 2023-09-29T22:22:50.469 DEBUG:boto:Token: None 2023-09-29T22:22:50.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee47c5ae1aeaf4d8-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.518 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.518 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.518 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.519 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.519 DEBUG:boto:Method: GET 2023-09-29T22:22:50.519 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.519 DEBUG:boto:Data: 2023-09-29T22:22:50.519 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.519 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.519 DEBUG:boto:Params: {} 2023-09-29T22:22:50.519 DEBUG:boto:Token: None 2023-09-29T22:22:50.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.520 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092071268d4796eae-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.522 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.522 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.522 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.522 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.522 DEBUG:boto:Method: GET 2023-09-29T22:22:50.522 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.522 DEBUG:boto:Data: 2023-09-29T22:22:50.522 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.523 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.523 DEBUG:boto:Params: {} 2023-09-29T22:22:50.523 DEBUG:boto:Token: None 2023-09-29T22:22:50.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.523 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ed6d8521b010e7-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.526 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.526 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.526 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.526 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.527 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.527 DEBUG:boto:Method: GET 2023-09-29T22:22:50.527 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.527 DEBUG:boto:Data: 2023-09-29T22:22:50.527 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.527 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.527 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.527 DEBUG:boto:Params: {} 2023-09-29T22:22:50.527 DEBUG:boto:Token: None 2023-09-29T22:22:50.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.528 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a25a26850fcc82d-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.530 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.531 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.531 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.531 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.531 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.531 DEBUG:boto:Method: GET 2023-09-29T22:22:50.531 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.531 DEBUG:boto:Data: 2023-09-29T22:22:50.531 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.532 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.532 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.532 DEBUG:boto:Params: {} 2023-09-29T22:22:50.532 DEBUG:boto:Token: None 2023-09-29T22:22:50.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4= 2023-09-29T22:22:50.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4=', 'Content-Length': '0'} 2023-09-29T22:22:50.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2a037077e7b72ab-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.534 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.534 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.534 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.534 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.535 DEBUG:boto:Method: GET 2023-09-29T22:22:50.535 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.535 DEBUG:boto:Data: 2023-09-29T22:22:50.535 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.535 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.535 DEBUG:boto:Params: {} 2023-09-29T22:22:50.535 DEBUG:boto:Token: None 2023-09-29T22:22:50.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4= 2023-09-29T22:22:50.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4=', 'Content-Length': '0'} 2023-09-29T22:22:50.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c23afe57d537a382-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.537 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.538 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.538 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.538 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.538 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.538 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.538 DEBUG:boto:Data: 2023-09-29T22:22:50.538 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.538 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.538 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.539 DEBUG:boto:Params: {} 2023-09-29T22:22:50.539 DEBUG:boto:Token: None 2023-09-29T22:22:50.539 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.539 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IKC2AmfDCNHNJClscKagjOew764= 2023-09-29T22:22:50.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IKC2AmfDCNHNJClscKagjOew764=', 'Content-Length': '0'} 2023-09-29T22:22:50.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a29687f4072d66f-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.541 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.541 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.541 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.541 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.542 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.542 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.542 DEBUG:boto:Data: 2023-09-29T22:22:50.542 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.542 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.542 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.542 DEBUG:boto:Params: {} 2023-09-29T22:22:50.542 DEBUG:boto:Token: None 2023-09-29T22:22:50.542 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.543 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IKC2AmfDCNHNJClscKagjOew764= 2023-09-29T22:22:50.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IKC2AmfDCNHNJClscKagjOew764=', 'Content-Length': '0'} 2023-09-29T22:22:50.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040adb33614ebb88c-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.544 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.545 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.545 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.545 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.545 DEBUG:boto:Method: GET 2023-09-29T22:22:50.545 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.545 DEBUG:boto:Data: 2023-09-29T22:22:50.545 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.545 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.545 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.545 DEBUG:boto:Params: {} 2023-09-29T22:22:50.546 DEBUG:boto:Token: None 2023-09-29T22:22:50.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.546 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4= 2023-09-29T22:22:50.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4=', 'Content-Length': '0'} 2023-09-29T22:22:50.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a26f7c0900b52968-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.548 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.548 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.548 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.548 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.548 DEBUG:boto:Method: GET 2023-09-29T22:22:50.548 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.549 DEBUG:boto:Data: 2023-09-29T22:22:50.549 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.549 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.549 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.549 DEBUG:boto:Params: {} 2023-09-29T22:22:50.549 DEBUG:boto:Token: None 2023-09-29T22:22:50.549 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:50.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4= 2023-09-29T22:22:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bvLSzcw6Nh+wfj61IemoYZSAfG4=', 'Content-Length': '0'} 2023-09-29T22:22:50.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058cbfb608c9a6d12-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.594 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.594 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.595 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.595 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.595 DEBUG:boto:Method: GET 2023-09-29T22:22:50.595 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.595 DEBUG:boto:Data: 2023-09-29T22:22:50.595 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.595 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.595 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.595 DEBUG:boto:Params: {} 2023-09-29T22:22:50.595 DEBUG:boto:Token: None 2023-09-29T22:22:50.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c1ec4c3bfcde56-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.598 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.599 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.599 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.599 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.599 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.599 DEBUG:boto:Method: GET 2023-09-29T22:22:50.599 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.599 DEBUG:boto:Data: 2023-09-29T22:22:50.599 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.599 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.600 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.600 DEBUG:boto:Params: {} 2023-09-29T22:22:50.600 DEBUG:boto:Token: None 2023-09-29T22:22:50.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.600 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943473f82e83d13f-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.602 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.603 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.603 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.603 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.603 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.603 DEBUG:boto:Data: 2023-09-29T22:22:50.603 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.603 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.604 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.604 DEBUG:boto:Params: {} 2023-09-29T22:22:50.604 DEBUG:boto:Token: None 2023-09-29T22:22:50.604 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013893f596117720f-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.607 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.607 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.607 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.607 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.607 DEBUG:boto:Data: 2023-09-29T22:22:50.608 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.608 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.608 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.608 DEBUG:boto:Params: {} 2023-09-29T22:22:50.608 DEBUG:boto:Token: None 2023-09-29T22:22:50.608 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.608 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8da8aa6042274e8-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.612 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.612 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.612 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.612 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.612 DEBUG:boto:Method: GET 2023-09-29T22:22:50.612 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.612 DEBUG:boto:Data: 2023-09-29T22:22:50.613 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.613 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.613 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.613 DEBUG:boto:Params: {} 2023-09-29T22:22:50.613 DEBUG:boto:Token: None 2023-09-29T22:22:50.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008db2d3e0f81fbeed-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.662 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.663 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.663 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.663 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.663 DEBUG:boto:Method: GET 2023-09-29T22:22:50.663 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.663 DEBUG:boto:Data: 2023-09-29T22:22:50.663 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.663 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.663 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.663 DEBUG:boto:Params: {} 2023-09-29T22:22:50.664 DEBUG:boto:Token: None 2023-09-29T22:22:50.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1c29bbb57f56ec2-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.666 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.666 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.666 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.667 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.667 DEBUG:boto:Method: GET 2023-09-29T22:22:50.667 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.667 DEBUG:boto:Data: 2023-09-29T22:22:50.667 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.667 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.667 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.667 DEBUG:boto:Params: {} 2023-09-29T22:22:50.667 DEBUG:boto:Token: None 2023-09-29T22:22:50.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18101e9f34860f7-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.670 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.671 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.671 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.671 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.671 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.671 DEBUG:boto:Method: GET 2023-09-29T22:22:50.671 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.671 DEBUG:boto:Data: 2023-09-29T22:22:50.671 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.672 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.672 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.672 DEBUG:boto:Params: {} 2023-09-29T22:22:50.672 DEBUG:boto:Token: None 2023-09-29T22:22:50.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095832bb4d74aa639-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.675 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.675 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.675 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.676 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.676 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.676 DEBUG:boto:Method: GET 2023-09-29T22:22:50.676 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.676 DEBUG:boto:Data: 2023-09-29T22:22:50.676 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.676 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.676 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.676 DEBUG:boto:Params: {} 2023-09-29T22:22:50.676 DEBUG:boto:Token: None 2023-09-29T22:22:50.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.677 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674= 2023-09-29T22:22:50.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674=', 'Content-Length': '0'} 2023-09-29T22:22:50.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6fad908a195d504-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.679 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.679 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.679 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.679 DEBUG:boto:Method: GET 2023-09-29T22:22:50.679 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.679 DEBUG:boto:Data: 2023-09-29T22:22:50.680 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.680 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.680 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.680 DEBUG:boto:Params: {} 2023-09-29T22:22:50.680 DEBUG:boto:Token: None 2023-09-29T22:22:50.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674= 2023-09-29T22:22:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674=', 'Content-Length': '0'} 2023-09-29T22:22:50.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2d559b45e9716e5-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.682 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.682 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.682 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.683 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.683 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.683 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.683 DEBUG:boto:Data: 2023-09-29T22:22:50.683 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.683 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.683 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.683 DEBUG:boto:Params: {} 2023-09-29T22:22:50.683 DEBUG:boto:Token: None 2023-09-29T22:22:50.684 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vIgmSZs619TaWLPVPYafHZba5LY= 2023-09-29T22:22:50.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vIgmSZs619TaWLPVPYafHZba5LY=', 'Content-Length': '0'} 2023-09-29T22:22:50.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df3607e1abaa62c6-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.686 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.686 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.686 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.686 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.686 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.686 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.686 DEBUG:boto:Data: 2023-09-29T22:22:50.686 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.687 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.687 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.687 DEBUG:boto:Params: {} 2023-09-29T22:22:50.687 DEBUG:boto:Token: None 2023-09-29T22:22:50.687 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vIgmSZs619TaWLPVPYafHZba5LY= 2023-09-29T22:22:50.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vIgmSZs619TaWLPVPYafHZba5LY=', 'Content-Length': '0'} 2023-09-29T22:22:50.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001fc8147c92538da-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.689 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.689 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.689 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.689 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.690 DEBUG:boto:Method: GET 2023-09-29T22:22:50.690 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.690 DEBUG:boto:Data: 2023-09-29T22:22:50.690 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.690 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.690 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.690 DEBUG:boto:Params: {} 2023-09-29T22:22:50.690 DEBUG:boto:Token: None 2023-09-29T22:22:50.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674= 2023-09-29T22:22:50.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674=', 'Content-Length': '0'} 2023-09-29T22:22:50.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c312c47b7dfb56c8-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.734 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.735 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.735 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.735 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.735 DEBUG:boto:Method: GET 2023-09-29T22:22:50.735 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.735 DEBUG:boto:Data: 2023-09-29T22:22:50.735 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.735 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.736 DEBUG:boto:Params: {} 2023-09-29T22:22:50.736 DEBUG:boto:Token: None 2023-09-29T22:22:50.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:50.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674= 2023-09-29T22:22:50.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSd6LY8ykuEbXKZ98R2VK/kd674=', 'Content-Length': '0'} 2023-09-29T22:22:50.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1b9b54d277836de-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.738 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.738 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.739 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.739 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.739 DEBUG:boto:Method: GET 2023-09-29T22:22:50.739 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.739 DEBUG:boto:Data: 2023-09-29T22:22:50.739 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.739 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.739 DEBUG:boto:Params: {} 2023-09-29T22:22:50.740 DEBUG:boto:Token: None 2023-09-29T22:22:50.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddb2671860907d5-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.743 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.743 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.743 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.743 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.743 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.744 DEBUG:boto:Method: GET 2023-09-29T22:22:50.744 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.744 DEBUG:boto:Data: 2023-09-29T22:22:50.744 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.744 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.744 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.744 DEBUG:boto:Params: {} 2023-09-29T22:22:50.744 DEBUG:boto:Token: None 2023-09-29T22:22:50.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007784995b66cfe9c9-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.747 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.747 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.748 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.748 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.748 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.748 DEBUG:boto:Data: 2023-09-29T22:22:50.748 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.748 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.748 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.748 DEBUG:boto:Params: {} 2023-09-29T22:22:50.748 DEBUG:boto:Token: None 2023-09-29T22:22:50.749 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.749 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ff9b2244456f0f8-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.752 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.752 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.752 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.752 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.752 DEBUG:boto:Data: 2023-09-29T22:22:50.753 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.753 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.753 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.753 DEBUG:boto:Params: {} 2023-09-29T22:22:50.753 DEBUG:boto:Token: None 2023-09-29T22:22:50.753 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.753 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026bc896f2cd4321d-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.757 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.757 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.757 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.757 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.757 DEBUG:boto:Method: GET 2023-09-29T22:22:50.757 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.757 DEBUG:boto:Data: 2023-09-29T22:22:50.757 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.757 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.758 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.758 DEBUG:boto:Params: {} 2023-09-29T22:22:50.758 DEBUG:boto:Token: None 2023-09-29T22:22:50.758 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.758 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000056291deff238ea2-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.802 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.802 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.803 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.803 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.803 DEBUG:boto:Method: GET 2023-09-29T22:22:50.803 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.803 DEBUG:boto:Data: 2023-09-29T22:22:50.803 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.803 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.803 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.803 DEBUG:boto:Params: {} 2023-09-29T22:22:50.804 DEBUG:boto:Token: None 2023-09-29T22:22:50.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.804 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ca0d1e191e5bd88-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.806 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.806 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.806 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.806 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.806 DEBUG:boto:Method: GET 2023-09-29T22:22:50.806 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.807 DEBUG:boto:Data: 2023-09-29T22:22:50.807 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.807 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.807 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.807 DEBUG:boto:Params: {} 2023-09-29T22:22:50.807 DEBUG:boto:Token: None 2023-09-29T22:22:50.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.807 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cfbacd8ea33c393-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.810 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.810 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.810 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.810 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.811 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.811 DEBUG:boto:Method: GET 2023-09-29T22:22:50.811 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.811 DEBUG:boto:Data: 2023-09-29T22:22:50.811 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.811 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.811 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.811 DEBUG:boto:Params: {} 2023-09-29T22:22:50.811 DEBUG:boto:Token: None 2023-09-29T22:22:50.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb1e0588539dcde-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.815 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.815 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.815 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.815 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.815 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.816 DEBUG:boto:Method: GET 2023-09-29T22:22:50.816 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.816 DEBUG:boto:Data: 2023-09-29T22:22:50.816 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.816 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.816 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.816 DEBUG:boto:Params: {} 2023-09-29T22:22:50.816 DEBUG:boto:Token: None 2023-09-29T22:22:50.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s= 2023-09-29T22:22:50.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s=', 'Content-Length': '0'} 2023-09-29T22:22:50.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b944cf2bc40b04f-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.819 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.819 DEBUG:boto:Method: GET 2023-09-29T22:22:50.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.819 DEBUG:boto:Data: 2023-09-29T22:22:50.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.820 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.820 DEBUG:boto:Params: {} 2023-09-29T22:22:50.820 DEBUG:boto:Token: None 2023-09-29T22:22:50.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s= 2023-09-29T22:22:50.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s=', 'Content-Length': '0'} 2023-09-29T22:22:50.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000952275213de864b3-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.862 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.862 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.863 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.863 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.863 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.863 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.863 DEBUG:boto:Data: 2023-09-29T22:22:50.863 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.863 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.863 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.863 DEBUG:boto:Params: {} 2023-09-29T22:22:50.863 DEBUG:boto:Token: None 2023-09-29T22:22:50.864 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DNOU0IJ6IR/9nsZwqmoQwhfk+kg= 2023-09-29T22:22:50.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DNOU0IJ6IR/9nsZwqmoQwhfk+kg=', 'Content-Length': '0'} 2023-09-29T22:22:50.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf60d4ec7ef5f711-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.866 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.866 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.866 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.866 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.866 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.866 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.866 DEBUG:boto:Data: 2023-09-29T22:22:50.866 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.867 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.867 DEBUG:boto:Params: {} 2023-09-29T22:22:50.867 DEBUG:boto:Token: None 2023-09-29T22:22:50.867 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DNOU0IJ6IR/9nsZwqmoQwhfk+kg= 2023-09-29T22:22:50.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DNOU0IJ6IR/9nsZwqmoQwhfk+kg=', 'Content-Length': '0'} 2023-09-29T22:22:50.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f4b3af0eaeabb30-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.869 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.869 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.869 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.869 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.869 DEBUG:boto:Method: GET 2023-09-29T22:22:50.870 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.870 DEBUG:boto:Data: 2023-09-29T22:22:50.870 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.870 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.870 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.870 DEBUG:boto:Params: {} 2023-09-29T22:22:50.870 DEBUG:boto:Token: None 2023-09-29T22:22:50.870 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s= 2023-09-29T22:22:50.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s=', 'Content-Length': '0'} 2023-09-29T22:22:50.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bece6bbec8e9d4a-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.914 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.915 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.915 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.915 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.915 DEBUG:boto:Method: GET 2023-09-29T22:22:50.915 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.915 DEBUG:boto:Data: 2023-09-29T22:22:50.915 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.915 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.915 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.915 DEBUG:boto:Params: {} 2023-09-29T22:22:50.916 DEBUG:boto:Token: None 2023-09-29T22:22:50.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:50.916 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s= 2023-09-29T22:22:50.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TkjY8A1/j8AFLZfWI7KR8CLKc6s=', 'Content-Length': '0'} 2023-09-29T22:22:50.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036c98028f2d4c258-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.958 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.958 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.959 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.959 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.959 DEBUG:boto:Method: GET 2023-09-29T22:22:50.959 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.959 DEBUG:boto:Data: 2023-09-29T22:22:50.959 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.959 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.959 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.959 DEBUG:boto:Params: {} 2023-09-29T22:22:50.960 DEBUG:boto:Token: None 2023-09-29T22:22:50.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d682a9e85eba833a-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.963 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.963 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:50.963 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:50.964 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.964 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.964 DEBUG:boto:Method: GET 2023-09-29T22:22:50.964 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:50.964 DEBUG:boto:Data: 2023-09-29T22:22:50.964 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.964 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.964 DEBUG:boto:Params: {} 2023-09-29T22:22:50.964 DEBUG:boto:Token: None 2023-09-29T22:22:50.965 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/ 2023-09-29T22:22:50.965 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI= 2023-09-29T22:22:50.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BGGKttggAj0/Eyri7pfAdD/DovI=', 'Content-Length': '0'} 2023-09-29T22:22:50.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004931618d6d32ed13-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.967 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:50.968 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.968 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.968 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.968 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.968 DEBUG:boto:Data: 2023-09-29T22:22:50.968 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.968 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.968 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.968 DEBUG:boto:Params: {} 2023-09-29T22:22:50.969 DEBUG:boto:Token: None 2023-09-29T22:22:50.969 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007790dd6eb13191f9-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.972 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.972 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.973 DEBUG:boto:Method: HEAD 2023-09-29T22:22:50.973 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:50.973 DEBUG:boto:Data: 2023-09-29T22:22:50.973 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.973 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:50.973 DEBUG:boto:Port: 8000 2023-09-29T22:22:50.973 DEBUG:boto:Params: {} 2023-09-29T22:22:50.973 DEBUG:boto:Token: None 2023-09-29T22:22:50.973 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj 2023-09-29T22:22:50.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI= 2023-09-29T22:22:50.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aRS2rD6vY46B0nrti4Be/7eODvI=', 'Content-Length': '0'} 2023-09-29T22:22:50.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2d3213adb0be2bc-0065174e3a-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:50.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:50.977 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:50.977 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:50.977 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.977 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.977 DEBUG:boto:Method: GET 2023-09-29T22:22:50.977 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.978 DEBUG:boto:Data: 2023-09-29T22:22:50.978 DEBUG:boto:Headers: {} 2023-09-29T22:22:50.978 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:50.978 DEBUG:boto:Port: 8001 2023-09-29T22:22:50.978 DEBUG:boto:Params: {} 2023-09-29T22:22:50.978 DEBUG:boto:Token: None 2023-09-29T22:22:50.978 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:50 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:50.978 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc= 2023-09-29T22:22:50.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:50 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BttFUCiKeNsbnAcoPSJiBTN9sHc=', 'Content-Length': '0'} 2023-09-29T22:22:50.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000434539c85e955768-0065174e3a-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.022 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.023 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.023 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.023 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.023 DEBUG:boto:Method: GET 2023-09-29T22:22:51.023 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.023 DEBUG:boto:Data: 2023-09-29T22:22:51.023 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.023 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:51.023 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.023 DEBUG:boto:Params: {} 2023-09-29T22:22:51.024 DEBUG:boto:Token: None 2023-09-29T22:22:51.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab774fb59e49a808-0065174e3b-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.026 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.026 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.026 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.027 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.027 DEBUG:boto:Method: GET 2023-09-29T22:22:51.027 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.027 DEBUG:boto:Data: 2023-09-29T22:22:51.027 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.027 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.027 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.027 DEBUG:boto:Params: {} 2023-09-29T22:22:51.027 DEBUG:boto:Token: None 2023-09-29T22:22:51.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c4047d6ef62dc9-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.031 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.031 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.031 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.031 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.031 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.032 DEBUG:boto:Method: GET 2023-09-29T22:22:51.033 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.033 DEBUG:boto:Data: 2023-09-29T22:22:51.033 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.033 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:22:51.033 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.033 DEBUG:boto:Params: {} 2023-09-29T22:22:51.034 DEBUG:boto:Token: None 2023-09-29T22:22:51.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ad9a5062288801-0065174e3b-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.036 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone3, test-zone1} 2023-09-29T22:22:51.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-19 zones={test-zone3, test-zone2} 2023-09-29T22:22:51.037 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.037 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.037 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.037 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:51.037 DEBUG:boto:Data: 2023-09-29T22:22:51.038 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.038 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.038 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.038 DEBUG:boto:Params: {} 2023-09-29T22:22:51.038 DEBUG:boto:Token: None 2023-09-29T22:22:51.038 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4kj+WxJDZbjq+g5+wg4mwwVh/pg= 2023-09-29T22:22:51.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4kj+WxJDZbjq+g5+wg4mwwVh/pg=', 'Content-Length': '0'} 2023-09-29T22:22:51.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7c9450a7e86146c-0065174e3b-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.041 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.041 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.041 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.041 DEBUG:boto:Path: /dmuhsu-19/ 2023-09-29T22:22:51.041 DEBUG:boto:Data: 2023-09-29T22:22:51.041 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.041 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.042 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.042 DEBUG:boto:Params: {} 2023-09-29T22:22:51.042 DEBUG:boto:Token: None 2023-09-29T22:22:51.042 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.042 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4kj+WxJDZbjq+g5+wg4mwwVh/pg= 2023-09-29T22:22:51.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4kj+WxJDZbjq+g5+wg4mwwVh/pg=', 'Content-Length': '0'} 2023-09-29T22:22:51.045 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000971a9d764c478e4a-0065174e3b-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.045 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:22:51.045 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.045 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.045 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.045 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.046 DEBUG:boto:Method: GET 2023-09-29T22:22:51.046 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.046 DEBUG:boto:Data: 2023-09-29T22:22:51.046 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.046 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.046 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.046 DEBUG:boto:Params: {} 2023-09-29T22:22:51.046 DEBUG:boto:Token: None 2023-09-29T22:22:51.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/?versions 2023-09-29T22:22:51.046 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU= 2023-09-29T22:22:51.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU=', 'Content-Length': '0'} 2023-09-29T22:22:51.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060735a1168706716-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.050 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.051 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.054 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:22:51.054 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.055 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.055 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.055 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.055 DEBUG:boto:Method: GET 2023-09-29T22:22:51.055 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.055 DEBUG:boto:Data: 2023-09-29T22:22:51.055 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.055 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.055 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.055 DEBUG:boto:Params: {} 2023-09-29T22:22:51.056 DEBUG:boto:Token: None 2023-09-29T22:22:51.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/?versions 2023-09-29T22:22:51.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU= 2023-09-29T22:22:51.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU=', 'Content-Length': '0'} 2023-09-29T22:22:51.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360cc54ba8a7a6b5-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.059 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:22:51.064 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.064 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.064 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.064 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.064 DEBUG:boto:Method: GET 2023-09-29T22:22:51.064 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.064 DEBUG:boto:Data: 2023-09-29T22:22:51.064 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.065 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.065 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.065 DEBUG:boto:Params: {} 2023-09-29T22:22:51.065 DEBUG:boto:Token: None 2023-09-29T22:22:51.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/?versions 2023-09-29T22:22:51.065 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU= 2023-09-29T22:22:51.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU=', 'Content-Length': '0'} 2023-09-29T22:22:51.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7bd6f59acc059bd-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.069 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.070 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.071 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.071 DEBUG:boto:path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.071 DEBUG:boto:auth_path=/dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.071 DEBUG:boto:Method: GET 2023-09-29T22:22:51.071 DEBUG:boto:Path: /dmuhsu-19/?versions&max-keys=999 2023-09-29T22:22:51.071 DEBUG:boto:Data: 2023-09-29T22:22:51.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.071 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.071 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.071 DEBUG:boto:Params: {} 2023-09-29T22:22:51.072 DEBUG:boto:Token: None 2023-09-29T22:22:51.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/?versions 2023-09-29T22:22:51.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU= 2023-09-29T22:22:51.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OvvfX1+fkfWe7cWahiEt4KASDdU=', 'Content-Length': '0'} 2023-09-29T22:22:51.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0c864f02596bfe5-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.075 DEBUG:boto:b'dmuhsu-19999falsedummyVmu.blizjrjgHMGAw4I-7Zxv9PsRnm5true2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlr-1f0YGIF5FSQ7y.7AlfCOIl94EQshtrue2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuaJvHIry8ucgM9SA-2KVFcA0Av3avB-false2023-09-29T22:19:59.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxQoyCkFIJBdsfuXIuPyk.0kVgu7C9edfalse2023-09-29T22:19:59.873Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zNfalse2023-09-29T22:19:59.779Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjH62A4zyj3HF6.PqOXDao2iSXg0kywxSfalse2023-09-29T22:19:59.794Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpHUoRnAhPF0mOKRAF4HQOhhBLX0yJHhfalse2023-09-29T22:19:59.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2MT28my.DVcM5Kd42w8oaZYB1cPftptfalse2023-09-29T22:19:59.719Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjARt2wNmFlEw7PolmrP0bT3B51j6z7oxfalse2023-09-29T22:19:59.621Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdbJpbtOestfBMgpk.e7uPehxzJXPIMHfalse2023-09-29T22:19:59.671Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhBI5xQc0xpClMdbPsqNZ0ltzLt4HItwfalse2023-09-29T22:19:59.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjut4FW9RW-2lP6ExrM4RvjthXiEpdQwdfalse2023-09-29T22:19:59.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJfalse2023-09-29T22:19:59.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhOfalse2023-09-29T22:19:59.513Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCwIobdwlPKtbIJWJbuSlt9ZJ6IK7upFfalse2023-09-29T22:19:59.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhg.3uHbA9vEsXztaQyye1kYMtpgUPV4false2023-09-29T22:19:59.526Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFfTbXU5TRCNthRqwusMgCllOl6aNgs9false2023-09-29T22:19:59.486Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtCfalse2023-09-29T22:19:59.437Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjjpJVcdJHvZoYOCaa43L2AAr7v0bG4-gfalse2023-09-29T22:19:59.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjHWlT9cq3Z9KfNRxOyf6NTGYD6.iFXkafalse2023-09-29T22:19:59.369Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJJO1acwgSpskwqQWN.dkCFtTPz5ryQjfalse2023-09-29T22:19:59.337Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjy84w72ok3GVudkBmY3Rlj4Nja225EFVfalse2023-09-29T22:19:59.354Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdu7slEJ9B4w6NbWPM.cHdXkeB9PhJlfalse2023-09-29T22:19:59.269Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjN0xc2mTHYugKUYrfwv6R6Qf.tHuMvtcfalse2023-09-29T22:19:59.284Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPTPlqPNXFgb6F3p0fEU-rPyLk8RXduCfalse2023-09-29T22:19:59.255Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5670vOkpV3J9eMlK7bQCZbnI9iPRJoxfalse2023-09-29T22:19:59.228Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKiXedrIktpOkLGXNWQzhwbI2HyTS7Kyfalse2023-09-29T22:19:59.242Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjP47GybDIgug9dC.sEibuF-MGXAmUiuafalse2023-09-29T22:19:59.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSScUnRPCrcSNC5QQL1tehNEyL0eXHiMfalse2023-09-29T22:19:59.161Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt6DBuD82EJPXFqtJn.Sul7SXH3eEr9wfalse2023-09-29T22:19:59.131Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzNfF-8Qv4tifWWzD8LieUTqh64w4TOkfalse2023-09-29T22:19:59.146Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:51.077 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.077 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.077 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.077 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.077 DEBUG:boto:Method: GET 2023-09-29T22:22:51.078 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.078 DEBUG:boto:Data: 2023-09-29T22:22:51.078 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.078 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.078 DEBUG:boto:Params: {} 2023-09-29T22:22:51.078 DEBUG:boto:Token: None 2023-09-29T22:22:51.078 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.078 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bec01f4773a651e-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.080 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.081 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.081 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.081 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.081 DEBUG:boto:Method: GET 2023-09-29T22:22:51.081 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.081 DEBUG:boto:Data: 2023-09-29T22:22:51.081 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.081 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.081 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.081 DEBUG:boto:Params: {} 2023-09-29T22:22:51.082 DEBUG:boto:Token: None 2023-09-29T22:22:51.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000069a293b4ba96479-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.084 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.084 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.084 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.084 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.084 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.085 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.085 DEBUG:boto:Data: 2023-09-29T22:22:51.085 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.085 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.085 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.085 DEBUG:boto:Params: {} 2023-09-29T22:22:51.085 DEBUG:boto:Token: None 2023-09-29T22:22:51.085 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.085 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SXr+ym1OIleYYCqhB47C+IDAGCQ= 2023-09-29T22:22:51.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SXr+ym1OIleYYCqhB47C+IDAGCQ=', 'Content-Length': '0'} 2023-09-29T22:22:51.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d53711c8d019da6-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.087 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.088 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.088 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.088 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.088 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.088 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.088 DEBUG:boto:Data: 2023-09-29T22:22:51.088 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.088 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.088 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.088 DEBUG:boto:Params: {} 2023-09-29T22:22:51.089 DEBUG:boto:Token: None 2023-09-29T22:22:51.089 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.089 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SXr+ym1OIleYYCqhB47C+IDAGCQ= 2023-09-29T22:22:51.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SXr+ym1OIleYYCqhB47C+IDAGCQ=', 'Content-Length': '0'} 2023-09-29T22:22:51.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000daa0ea0f0a4cc614-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:51.091 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.091 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.091 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.091 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.091 DEBUG:boto:Method: GET 2023-09-29T22:22:51.091 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.092 DEBUG:boto:Data: 2023-09-29T22:22:51.092 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.092 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.092 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.092 DEBUG:boto:Params: {} 2023-09-29T22:22:51.092 DEBUG:boto:Token: None 2023-09-29T22:22:51.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.092 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000754cf59d49f05897-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.094 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.095 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.095 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.095 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.095 DEBUG:boto:Method: GET 2023-09-29T22:22:51.095 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.095 DEBUG:boto:Data: 2023-09-29T22:22:51.095 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.095 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.095 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.095 DEBUG:boto:Params: {} 2023-09-29T22:22:51.096 DEBUG:boto:Token: None 2023-09-29T22:22:51.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ccf783d7e81a012-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.098 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.098 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.098 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.098 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.098 DEBUG:boto:Method: GET 2023-09-29T22:22:51.099 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.099 DEBUG:boto:Data: 2023-09-29T22:22:51.099 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.099 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.099 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.099 DEBUG:boto:Params: {} 2023-09-29T22:22:51.099 DEBUG:boto:Token: None 2023-09-29T22:22:51.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.099 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aded16b09941034-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.142 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.143 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.143 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.143 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.143 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.143 DEBUG:boto:Method: GET 2023-09-29T22:22:51.143 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.143 DEBUG:boto:Data: 2023-09-29T22:22:51.143 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.143 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.143 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.144 DEBUG:boto:Params: {} 2023-09-29T22:22:51.144 DEBUG:boto:Token: None 2023-09-29T22:22:51.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a3206ff2c1b8a6-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.147 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.147 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.147 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.147 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.147 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:51.147 DEBUG:boto:Data: 2023-09-29T22:22:51.148 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.148 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.148 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.148 DEBUG:boto:Params: {} 2023-09-29T22:22:51.148 DEBUG:boto:Token: None 2023-09-29T22:22:51.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy 2023-09-29T22:22:51.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qxiR3VXefaul+nx6iOy+9egPSNo= 2023-09-29T22:22:51.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qxiR3VXefaul+nx6iOy+9egPSNo=', 'Content-Length': '0'} 2023-09-29T22:22:51.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051267e2e27b35d21-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.151 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.151 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.151 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.151 DEBUG:boto:Path: /dmuhsu-19/dummy 2023-09-29T22:22:51.151 DEBUG:boto:Data: 2023-09-29T22:22:51.151 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.151 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.151 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.152 DEBUG:boto:Params: {} 2023-09-29T22:22:51.152 DEBUG:boto:Token: None 2023-09-29T22:22:51.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy 2023-09-29T22:22:51.152 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qxiR3VXefaul+nx6iOy+9egPSNo= 2023-09-29T22:22:51.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qxiR3VXefaul+nx6iOy+9egPSNo=', 'Content-Length': '0'} 2023-09-29T22:22:51.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000958cb2cf3a2e436c-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:22:51.155 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.155 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.155 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.155 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.155 DEBUG:boto:Method: GET 2023-09-29T22:22:51.155 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.156 DEBUG:boto:Data: 2023-09-29T22:22:51.156 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.156 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.156 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.156 DEBUG:boto:Params: {} 2023-09-29T22:22:51.156 DEBUG:boto:Token: None 2023-09-29T22:22:51.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.156 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b68781559a5ceba1-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.158 DEBUG:boto:path=/dmuhsu-19/dummy 2023-09-29T22:22:51.159 DEBUG:boto:auth_path=/dmuhsu-19/dummy 2023-09-29T22:22:51.159 DEBUG:boto:path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.159 DEBUG:boto:auth_path=/dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.159 DEBUG:boto:Method: GET 2023-09-29T22:22:51.159 DEBUG:boto:Path: /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.159 DEBUG:boto:Data: 2023-09-29T22:22:51.159 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.159 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.159 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.159 DEBUG:boto:Params: {} 2023-09-29T22:22:51.160 DEBUG:boto:Token: None 2023-09-29T22:22:51.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/dummy?versionId=Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5 2023-09-29T22:22:51.160 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI= 2023-09-29T22:22:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1NrdHiLNXu7I76zu8E3/DqkVDI=', 'Content-Length': '0'} 2023-09-29T22:22:51.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:27 GMT'), ('x-amz-version-id', 'Vmu.blizjrjgHMGAw4I-7Zxv9PsRnm5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c043ddf44e5acae-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.162 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.162 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.162 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.162 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.162 DEBUG:boto:Method: GET 2023-09-29T22:22:51.162 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.162 DEBUG:boto:Data: 2023-09-29T22:22:51.163 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.163 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.163 DEBUG:boto:Params: {} 2023-09-29T22:22:51.163 DEBUG:boto:Token: None 2023-09-29T22:22:51.163 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.163 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9760771e1f037f7-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.166 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.166 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.166 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.167 DEBUG:boto:path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.167 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.167 DEBUG:boto:Method: GET 2023-09-29T22:22:51.167 DEBUG:boto:Path: /dmuhsu-19/?prefix=dummy 2023-09-29T22:22:51.167 DEBUG:boto:Data: 2023-09-29T22:22:51.167 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.167 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.167 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.167 DEBUG:boto:Params: {} 2023-09-29T22:22:51.167 DEBUG:boto:Token: None 2023-09-29T22:22:51.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae279e0a9d7c874-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.170 DEBUG:boto:b'dmuhsu-19dummy1000falsedummy2023-09-29T22:19:27.928Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.171 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.171 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.171 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.171 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.171 DEBUG:boto:Method: GET 2023-09-29T22:22:51.171 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.171 DEBUG:boto:Data: 2023-09-29T22:22:51.171 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.171 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.171 DEBUG:boto:Params: {} 2023-09-29T22:22:51.172 DEBUG:boto:Token: None 2023-09-29T22:22:51.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008804355a788712b8-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.174 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.174 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.174 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.174 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.174 DEBUG:boto:Method: GET 2023-09-29T22:22:51.174 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.174 DEBUG:boto:Data: 2023-09-29T22:22:51.175 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.175 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.175 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.175 DEBUG:boto:Params: {} 2023-09-29T22:22:51.175 DEBUG:boto:Token: None 2023-09-29T22:22:51.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd3f67fea68f629d-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.218 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.218 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.218 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.219 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.219 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.219 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.219 DEBUG:boto:Data: 2023-09-29T22:22:51.219 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.219 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.219 DEBUG:boto:Params: {} 2023-09-29T22:22:51.219 DEBUG:boto:Token: None 2023-09-29T22:22:51.219 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfhHDdlvF5PvGHY/tk8zKcJNtng= 2023-09-29T22:22:51.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfhHDdlvF5PvGHY/tk8zKcJNtng=', 'Content-Length': '0'} 2023-09-29T22:22:51.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c627da45ac44049-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.221 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.222 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.222 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.222 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.222 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.222 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.222 DEBUG:boto:Data: 2023-09-29T22:22:51.222 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.222 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.222 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.222 DEBUG:boto:Params: {} 2023-09-29T22:22:51.223 DEBUG:boto:Token: None 2023-09-29T22:22:51.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZfhHDdlvF5PvGHY/tk8zKcJNtng= 2023-09-29T22:22:51.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZfhHDdlvF5PvGHY/tk8zKcJNtng=', 'Content-Length': '0'} 2023-09-29T22:22:51.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aea9f4c775659884-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.225 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.225 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.225 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.225 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.225 DEBUG:boto:Method: GET 2023-09-29T22:22:51.225 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.226 DEBUG:boto:Data: 2023-09-29T22:22:51.226 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.226 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.226 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.226 DEBUG:boto:Params: {} 2023-09-29T22:22:51.226 DEBUG:boto:Token: None 2023-09-29T22:22:51.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085cb61053f71ef39-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.270 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.270 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.271 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.271 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.271 DEBUG:boto:Method: GET 2023-09-29T22:22:51.271 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.271 DEBUG:boto:Data: 2023-09-29T22:22:51.271 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.271 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.271 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.271 DEBUG:boto:Params: {} 2023-09-29T22:22:51.271 DEBUG:boto:Token: None 2023-09-29T22:22:51.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb494d01c59b67f0-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.314 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.314 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.315 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.315 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.315 DEBUG:boto:Method: GET 2023-09-29T22:22:51.315 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.315 DEBUG:boto:Data: 2023-09-29T22:22:51.315 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.315 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.315 DEBUG:boto:Params: {} 2023-09-29T22:22:51.315 DEBUG:boto:Token: None 2023-09-29T22:22:51.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e306d4ef1ef39517-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.362 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.363 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.363 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.363 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.363 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.363 DEBUG:boto:Method: GET 2023-09-29T22:22:51.363 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.363 DEBUG:boto:Data: 2023-09-29T22:22:51.363 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.363 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.364 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.364 DEBUG:boto:Params: {} 2023-09-29T22:22:51.364 DEBUG:boto:Token: None 2023-09-29T22:22:51.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.364 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b04571367e36d1-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.367 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.367 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.367 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.367 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.368 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.368 DEBUG:boto:Data: 2023-09-29T22:22:51.368 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.368 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.368 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.368 DEBUG:boto:Params: {} 2023-09-29T22:22:51.368 DEBUG:boto:Token: None 2023-09-29T22:22:51.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.368 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000853309a51eb28cdd-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.372 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.372 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.372 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.372 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.372 DEBUG:boto:Data: 2023-09-29T22:22:51.372 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.372 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.372 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.372 DEBUG:boto:Params: {} 2023-09-29T22:22:51.373 DEBUG:boto:Token: None 2023-09-29T22:22:51.373 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.373 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008296848cc558879a-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.376 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.376 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.376 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.377 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.377 DEBUG:boto:Method: GET 2023-09-29T22:22:51.377 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.377 DEBUG:boto:Data: 2023-09-29T22:22:51.377 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.377 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.377 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.377 DEBUG:boto:Params: {} 2023-09-29T22:22:51.377 DEBUG:boto:Token: None 2023-09-29T22:22:51.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.378 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c42b1fccc5c48c5-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.422 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.422 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.423 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.423 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.423 DEBUG:boto:Method: GET 2023-09-29T22:22:51.423 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.423 DEBUG:boto:Data: 2023-09-29T22:22:51.423 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.423 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.423 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.423 DEBUG:boto:Params: {} 2023-09-29T22:22:51.424 DEBUG:boto:Token: None 2023-09-29T22:22:51.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085dd1411d32f1668-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.470 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.471 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.471 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.471 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.471 DEBUG:boto:Method: GET 2023-09-29T22:22:51.471 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.471 DEBUG:boto:Data: 2023-09-29T22:22:51.471 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.471 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.471 DEBUG:boto:Params: {} 2023-09-29T22:22:51.472 DEBUG:boto:Token: None 2023-09-29T22:22:51.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac7f907523ce5ef2-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.522 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.523 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.523 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.523 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.523 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.523 DEBUG:boto:Method: GET 2023-09-29T22:22:51.523 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.523 DEBUG:boto:Data: 2023-09-29T22:22:51.523 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.524 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.524 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.524 DEBUG:boto:Params: {} 2023-09-29T22:22:51.524 DEBUG:boto:Token: None 2023-09-29T22:22:51.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6ee6a04c159c6b-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.527 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.528 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.528 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.528 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.528 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.528 DEBUG:boto:Method: GET 2023-09-29T22:22:51.528 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.528 DEBUG:boto:Data: 2023-09-29T22:22:51.528 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.529 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.529 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.529 DEBUG:boto:Params: {} 2023-09-29T22:22:51.529 DEBUG:boto:Token: None 2023-09-29T22:22:51.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.529 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY= 2023-09-29T22:22:51.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY=', 'Content-Length': '0'} 2023-09-29T22:22:51.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048c796bb2bd4944d-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.531 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.532 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.532 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.532 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.532 DEBUG:boto:Method: GET 2023-09-29T22:22:51.532 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.532 DEBUG:boto:Data: 2023-09-29T22:22:51.532 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.532 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.532 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.532 DEBUG:boto:Params: {} 2023-09-29T22:22:51.533 DEBUG:boto:Token: None 2023-09-29T22:22:51.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.533 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY= 2023-09-29T22:22:51.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY=', 'Content-Length': '0'} 2023-09-29T22:22:51.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9cbb06e302cfb55-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.535 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.535 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.535 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.535 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.535 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.535 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.536 DEBUG:boto:Data: 2023-09-29T22:22:51.536 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.536 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.536 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.536 DEBUG:boto:Params: {} 2023-09-29T22:22:51.536 DEBUG:boto:Token: None 2023-09-29T22:22:51.536 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.536 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sNd92zOu981jh3X/JB0JDiLZhFo= 2023-09-29T22:22:51.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sNd92zOu981jh3X/JB0JDiLZhFo=', 'Content-Length': '0'} 2023-09-29T22:22:51.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000973b110370ca1c2d-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.538 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.539 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.539 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.539 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.539 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.539 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.539 DEBUG:boto:Data: 2023-09-29T22:22:51.539 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.539 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.539 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.539 DEBUG:boto:Params: {} 2023-09-29T22:22:51.540 DEBUG:boto:Token: None 2023-09-29T22:22:51.540 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sNd92zOu981jh3X/JB0JDiLZhFo= 2023-09-29T22:22:51.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sNd92zOu981jh3X/JB0JDiLZhFo=', 'Content-Length': '0'} 2023-09-29T22:22:51.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002834e1292e81bd63-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.542 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.542 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.542 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.542 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.542 DEBUG:boto:Method: GET 2023-09-29T22:22:51.543 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.543 DEBUG:boto:Data: 2023-09-29T22:22:51.543 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.543 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.543 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.543 DEBUG:boto:Params: {} 2023-09-29T22:22:51.543 DEBUG:boto:Token: None 2023-09-29T22:22:51.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.543 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY= 2023-09-29T22:22:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY=', 'Content-Length': '0'} 2023-09-29T22:22:51.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c15b303b8e0dba85-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.586 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.586 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.587 DEBUG:boto:path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.587 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.587 DEBUG:boto:Method: GET 2023-09-29T22:22:51.587 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.587 DEBUG:boto:Data: 2023-09-29T22:22:51.587 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.587 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.587 DEBUG:boto:Params: {} 2023-09-29T22:22:51.587 DEBUG:boto:Token: None 2023-09-29T22:22:51.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=uaJvHIry8ucgM9SA-2KVFcA0Av3avB- 2023-09-29T22:22:51.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY= 2023-09-29T22:22:51.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G0Hha9azE0n6ATAvOsOPYQ5fCdY=', 'Content-Length': '0'} 2023-09-29T22:22:51.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'uaJvHIry8ucgM9SA-2KVFcA0Av3avB-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016222d83e32c0c82-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.590 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.590 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.590 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.590 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.590 DEBUG:boto:Method: GET 2023-09-29T22:22:51.590 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.590 DEBUG:boto:Data: 2023-09-29T22:22:51.590 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.591 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.591 DEBUG:boto:Params: {} 2023-09-29T22:22:51.591 DEBUG:boto:Token: None 2023-09-29T22:22:51.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007abca4ee82ef7e94-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.634 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.635 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.635 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.635 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.635 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.635 DEBUG:boto:Method: GET 2023-09-29T22:22:51.635 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.635 DEBUG:boto:Data: 2023-09-29T22:22:51.635 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.635 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.635 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.636 DEBUG:boto:Params: {} 2023-09-29T22:22:51.636 DEBUG:boto:Token: None 2023-09-29T22:22:51.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.636 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4d9b2daedad8c9-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.639 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.639 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.639 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.639 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.639 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.639 DEBUG:boto:Data: 2023-09-29T22:22:51.640 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.640 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.640 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.640 DEBUG:boto:Params: {} 2023-09-29T22:22:51.640 DEBUG:boto:Token: None 2023-09-29T22:22:51.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.640 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003556730ba7dcddd4-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.644 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.644 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.644 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.644 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.644 DEBUG:boto:Data: 2023-09-29T22:22:51.644 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.644 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.644 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.644 DEBUG:boto:Params: {} 2023-09-29T22:22:51.644 DEBUG:boto:Token: None 2023-09-29T22:22:51.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.645 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001574378eac0693fe-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.648 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.648 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.648 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.648 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.648 DEBUG:boto:Method: GET 2023-09-29T22:22:51.649 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.649 DEBUG:boto:Data: 2023-09-29T22:22:51.649 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.649 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.649 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.649 DEBUG:boto:Params: {} 2023-09-29T22:22:51.649 DEBUG:boto:Token: None 2023-09-29T22:22:51.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.649 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099c122fb46bd3874-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.694 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.695 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.695 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.695 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.695 DEBUG:boto:Method: GET 2023-09-29T22:22:51.695 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.695 DEBUG:boto:Data: 2023-09-29T22:22:51.695 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.695 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.695 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.695 DEBUG:boto:Params: {} 2023-09-29T22:22:51.696 DEBUG:boto:Token: None 2023-09-29T22:22:51.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.696 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cd6d62625afa7cd-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.738 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.739 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.739 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.739 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.739 DEBUG:boto:Method: GET 2023-09-29T22:22:51.739 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.739 DEBUG:boto:Data: 2023-09-29T22:22:51.739 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.739 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.740 DEBUG:boto:Params: {} 2023-09-29T22:22:51.740 DEBUG:boto:Token: None 2023-09-29T22:22:51.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060d12efa87836f76-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.786 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.787 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.787 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.787 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.787 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.787 DEBUG:boto:Method: GET 2023-09-29T22:22:51.787 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.787 DEBUG:boto:Data: 2023-09-29T22:22:51.788 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.788 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.788 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.788 DEBUG:boto:Params: {} 2023-09-29T22:22:51.788 DEBUG:boto:Token: None 2023-09-29T22:22:51.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb37abb31af7499d-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.791 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.791 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.791 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.791 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.792 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.792 DEBUG:boto:Method: GET 2023-09-29T22:22:51.792 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.792 DEBUG:boto:Data: 2023-09-29T22:22:51.792 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.792 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.792 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.792 DEBUG:boto:Params: {} 2023-09-29T22:22:51.792 DEBUG:boto:Token: None 2023-09-29T22:22:51.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.793 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w= 2023-09-29T22:22:51.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w=', 'Content-Length': '0'} 2023-09-29T22:22:51.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d72f78f37eb125f-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.795 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.795 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.795 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.795 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.795 DEBUG:boto:Method: GET 2023-09-29T22:22:51.795 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.795 DEBUG:boto:Data: 2023-09-29T22:22:51.795 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.796 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.796 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.796 DEBUG:boto:Params: {} 2023-09-29T22:22:51.796 DEBUG:boto:Token: None 2023-09-29T22:22:51.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.796 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w= 2023-09-29T22:22:51.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w=', 'Content-Length': '0'} 2023-09-29T22:22:51.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a895c4eb5f1228b0-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.798 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.798 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.798 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.798 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.799 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.799 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.799 DEBUG:boto:Data: 2023-09-29T22:22:51.799 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.799 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.799 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.799 DEBUG:boto:Params: {} 2023-09-29T22:22:51.799 DEBUG:boto:Token: None 2023-09-29T22:22:51.799 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VhWcwncAG93SIvtk9/6iAw9umk8= 2023-09-29T22:22:51.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VhWcwncAG93SIvtk9/6iAw9umk8=', 'Content-Length': '0'} 2023-09-29T22:22:51.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b8ebdd9226aa24b-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.801 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.802 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.802 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.802 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.802 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.802 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.802 DEBUG:boto:Data: 2023-09-29T22:22:51.802 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.802 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.802 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.802 DEBUG:boto:Params: {} 2023-09-29T22:22:51.803 DEBUG:boto:Token: None 2023-09-29T22:22:51.803 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.803 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VhWcwncAG93SIvtk9/6iAw9umk8= 2023-09-29T22:22:51.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VhWcwncAG93SIvtk9/6iAw9umk8=', 'Content-Length': '0'} 2023-09-29T22:22:51.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000667b0e2317cfb07a-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.805 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.805 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.805 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.805 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.805 DEBUG:boto:Method: GET 2023-09-29T22:22:51.806 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.806 DEBUG:boto:Data: 2023-09-29T22:22:51.806 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.806 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.806 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.806 DEBUG:boto:Params: {} 2023-09-29T22:22:51.806 DEBUG:boto:Token: None 2023-09-29T22:22:51.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w= 2023-09-29T22:22:51.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w=', 'Content-Length': '0'} 2023-09-29T22:22:51.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d2db03a1063a276-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.808 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.809 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.809 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.809 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.809 DEBUG:boto:Method: GET 2023-09-29T22:22:51.809 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.809 DEBUG:boto:Data: 2023-09-29T22:22:51.809 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.809 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.809 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.809 DEBUG:boto:Params: {} 2023-09-29T22:22:51.810 DEBUG:boto:Token: None 2023-09-29T22:22:51.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed 2023-09-29T22:22:51.810 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w= 2023-09-29T22:22:51.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2iSy3LOggUlvj32H8UdEUmzES2w=', 'Content-Length': '0'} 2023-09-29T22:22:51.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xQoyCkFIJBdsfuXIuPyk.0kVgu7C9ed'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e77ccb2bc623ca6-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.854 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.855 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.855 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.855 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.855 DEBUG:boto:Method: GET 2023-09-29T22:22:51.855 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.855 DEBUG:boto:Data: 2023-09-29T22:22:51.855 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.855 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.855 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.855 DEBUG:boto:Params: {} 2023-09-29T22:22:51.856 DEBUG:boto:Token: None 2023-09-29T22:22:51.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039124dea5d2a431f-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.859 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.859 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:51.859 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:51.859 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.860 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.860 DEBUG:boto:Method: GET 2023-09-29T22:22:51.860 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:51.860 DEBUG:boto:Data: 2023-09-29T22:22:51.860 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.860 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.860 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.860 DEBUG:boto:Params: {} 2023-09-29T22:22:51.860 DEBUG:boto:Token: None 2023-09-29T22:22:51.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/ 2023-09-29T22:22:51.861 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE= 2023-09-29T22:22:51.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yATJpbDqqo4IVLBqV2noJwqMJTE=', 'Content-Length': '0'} 2023-09-29T22:22:51.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ff0e46e20a85c9-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.906 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:51.907 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.907 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.907 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.907 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.907 DEBUG:boto:Data: 2023-09-29T22:22:51.907 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.907 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.908 DEBUG:boto:Params: {} 2023-09-29T22:22:51.908 DEBUG:boto:Token: None 2023-09-29T22:22:51.908 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d95f251485926a6-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.911 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.911 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.912 DEBUG:boto:Method: HEAD 2023-09-29T22:22:51.912 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:51.912 DEBUG:boto:Data: 2023-09-29T22:22:51.912 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.912 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.912 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.912 DEBUG:boto:Params: {} 2023-09-29T22:22:51.912 DEBUG:boto:Token: None 2023-09-29T22:22:51.912 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj 2023-09-29T22:22:51.913 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk= 2023-09-29T22:22:51.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GW7GBTRsd/vQmRXhl0Kpt561wZk=', 'Content-Length': '0'} 2023-09-29T22:22:51.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000240a15ce14135fcf-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:51.916 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.916 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.916 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.916 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.916 DEBUG:boto:Method: GET 2023-09-29T22:22:51.917 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.917 DEBUG:boto:Data: 2023-09-29T22:22:51.917 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.917 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:51.917 DEBUG:boto:Port: 8001 2023-09-29T22:22:51.917 DEBUG:boto:Params: {} 2023-09-29T22:22:51.917 DEBUG:boto:Token: None 2023-09-29T22:22:51.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.917 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001524d68ef67189ac-0065174e3b-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:51.962 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:51.962 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:51.963 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.963 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.963 DEBUG:boto:Method: GET 2023-09-29T22:22:51.963 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.963 DEBUG:boto:Data: 2023-09-29T22:22:51.963 DEBUG:boto:Headers: {} 2023-09-29T22:22:51.963 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:51.963 DEBUG:boto:Port: 8000 2023-09-29T22:22:51.963 DEBUG:boto:Params: {} 2023-09-29T22:22:51.963 DEBUG:boto:Token: None 2023-09-29T22:22:51.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:51 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:51.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA= 2023-09-29T22:22:51.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:51 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:36mz+JWj8HJ8gJOAMw0RvlXlLxA=', 'Content-Length': '0'} 2023-09-29T22:22:51.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ff5a3084f790de-0065174e3b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.006 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.007 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.007 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.007 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.007 DEBUG:boto:Method: GET 2023-09-29T22:22:52.007 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.007 DEBUG:boto:Data: 2023-09-29T22:22:52.007 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.007 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.007 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.007 DEBUG:boto:Params: {} 2023-09-29T22:22:52.008 DEBUG:boto:Token: None 2023-09-29T22:22:52.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.008 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006698a108428755ca-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.011 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.011 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.011 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.011 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.011 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.011 DEBUG:boto:Method: GET 2023-09-29T22:22:52.011 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.011 DEBUG:boto:Data: 2023-09-29T22:22:52.012 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.012 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.012 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.012 DEBUG:boto:Params: {} 2023-09-29T22:22:52.012 DEBUG:boto:Token: None 2023-09-29T22:22:52.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.012 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003119805a3e0e1f52-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.058 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.059 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.059 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.059 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.059 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.059 DEBUG:boto:Method: GET 2023-09-29T22:22:52.059 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.059 DEBUG:boto:Data: 2023-09-29T22:22:52.059 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.060 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.060 DEBUG:boto:Params: {} 2023-09-29T22:22:52.060 DEBUG:boto:Token: None 2023-09-29T22:22:52.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E= 2023-09-29T22:22:52.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E=', 'Content-Length': '0'} 2023-09-29T22:22:52.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbcf3838d3ae394f-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.063 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.063 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.063 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.063 DEBUG:boto:Method: GET 2023-09-29T22:22:52.063 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.063 DEBUG:boto:Data: 2023-09-29T22:22:52.063 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.064 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.064 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.064 DEBUG:boto:Params: {} 2023-09-29T22:22:52.064 DEBUG:boto:Token: None 2023-09-29T22:22:52.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E= 2023-09-29T22:22:52.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E=', 'Content-Length': '0'} 2023-09-29T22:22:52.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aae9f3621f2e3cd4-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.066 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.066 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.066 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.067 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.067 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.067 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.067 DEBUG:boto:Data: 2023-09-29T22:22:52.067 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.067 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.067 DEBUG:boto:Params: {} 2023-09-29T22:22:52.067 DEBUG:boto:Token: None 2023-09-29T22:22:52.067 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VQpomtPzYjR9F/pT1QNKJcxPmAQ= 2023-09-29T22:22:52.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VQpomtPzYjR9F/pT1QNKJcxPmAQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8a725a9a89a4fdc-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.070 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.070 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.070 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.070 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.070 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.070 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.070 DEBUG:boto:Data: 2023-09-29T22:22:52.070 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.070 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.070 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.071 DEBUG:boto:Params: {} 2023-09-29T22:22:52.071 DEBUG:boto:Token: None 2023-09-29T22:22:52.071 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.071 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VQpomtPzYjR9F/pT1QNKJcxPmAQ= 2023-09-29T22:22:52.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VQpomtPzYjR9F/pT1QNKJcxPmAQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006301bbaa818a2c95-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.073 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.073 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.073 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.073 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.073 DEBUG:boto:Method: GET 2023-09-29T22:22:52.073 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.074 DEBUG:boto:Data: 2023-09-29T22:22:52.074 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.074 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.074 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.074 DEBUG:boto:Params: {} 2023-09-29T22:22:52.074 DEBUG:boto:Token: None 2023-09-29T22:22:52.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.074 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E= 2023-09-29T22:22:52.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E=', 'Content-Length': '0'} 2023-09-29T22:22:52.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000577ec6c3aa5d2100-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.118 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.118 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.118 DEBUG:boto:path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.119 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.119 DEBUG:boto:Method: GET 2023-09-29T22:22:52.119 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.119 DEBUG:boto:Data: 2023-09-29T22:22:52.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.119 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.119 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.119 DEBUG:boto:Params: {} 2023-09-29T22:22:52.119 DEBUG:boto:Token: None 2023-09-29T22:22:52.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN 2023-09-29T22:22:52.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E= 2023-09-29T22:22:52.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOlh1L2EIANfKsiGDSBCHnV8g9E=', 'Content-Length': '0'} 2023-09-29T22:22:52.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '6PkQmq.xAH4r.Ekecu2Kq8CRaMr90zN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0037e19aa8c419a-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.122 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.122 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.122 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.122 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.122 DEBUG:boto:Method: GET 2023-09-29T22:22:52.123 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.123 DEBUG:boto:Data: 2023-09-29T22:22:52.123 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.123 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.123 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.123 DEBUG:boto:Params: {} 2023-09-29T22:22:52.123 DEBUG:boto:Token: None 2023-09-29T22:22:52.123 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.123 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7455f30f9c766ef-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.170 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.171 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.171 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.171 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.171 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.171 DEBUG:boto:Method: GET 2023-09-29T22:22:52.171 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.171 DEBUG:boto:Data: 2023-09-29T22:22:52.172 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.172 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.172 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.172 DEBUG:boto:Params: {} 2023-09-29T22:22:52.172 DEBUG:boto:Token: None 2023-09-29T22:22:52.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2418d133226d83c-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.175 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.176 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.176 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.176 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.176 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.176 DEBUG:boto:Data: 2023-09-29T22:22:52.176 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.176 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.176 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.176 DEBUG:boto:Params: {} 2023-09-29T22:22:52.176 DEBUG:boto:Token: None 2023-09-29T22:22:52.177 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.177 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000646aaf211e7631c8-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.180 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.180 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.180 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.180 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.180 DEBUG:boto:Data: 2023-09-29T22:22:52.181 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.181 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.181 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.181 DEBUG:boto:Params: {} 2023-09-29T22:22:52.181 DEBUG:boto:Token: None 2023-09-29T22:22:52.181 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.181 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5896f202d9a363c-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.185 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.185 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.185 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.185 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.185 DEBUG:boto:Method: GET 2023-09-29T22:22:52.185 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.185 DEBUG:boto:Data: 2023-09-29T22:22:52.185 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.185 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.186 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.186 DEBUG:boto:Params: {} 2023-09-29T22:22:52.186 DEBUG:boto:Token: None 2023-09-29T22:22:52.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.186 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f72ac714d6411abc-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.230 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.230 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.230 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.231 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.231 DEBUG:boto:Method: GET 2023-09-29T22:22:52.231 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.231 DEBUG:boto:Data: 2023-09-29T22:22:52.231 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.231 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.231 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.231 DEBUG:boto:Params: {} 2023-09-29T22:22:52.231 DEBUG:boto:Token: None 2023-09-29T22:22:52.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002681d55b91b629be-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.278 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.279 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.279 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.279 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.279 DEBUG:boto:Method: GET 2023-09-29T22:22:52.279 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.279 DEBUG:boto:Data: 2023-09-29T22:22:52.279 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.279 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.279 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.279 DEBUG:boto:Params: {} 2023-09-29T22:22:52.280 DEBUG:boto:Token: None 2023-09-29T22:22:52.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.280 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005596a0b151c6a632-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.326 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.327 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.327 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.327 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.327 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.327 DEBUG:boto:Method: GET 2023-09-29T22:22:52.327 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.327 DEBUG:boto:Data: 2023-09-29T22:22:52.328 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.328 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.328 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.328 DEBUG:boto:Params: {} 2023-09-29T22:22:52.328 DEBUG:boto:Token: None 2023-09-29T22:22:52.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534d99fe4a3d1b1c-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.331 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.331 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.331 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.332 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.332 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.332 DEBUG:boto:Method: GET 2023-09-29T22:22:52.332 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.332 DEBUG:boto:Data: 2023-09-29T22:22:52.332 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.332 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.332 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.332 DEBUG:boto:Params: {} 2023-09-29T22:22:52.332 DEBUG:boto:Token: None 2023-09-29T22:22:52.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E= 2023-09-29T22:22:52.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E=', 'Content-Length': '0'} 2023-09-29T22:22:52.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c326c25b3eb7607-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.335 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.335 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.335 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.335 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.335 DEBUG:boto:Method: GET 2023-09-29T22:22:52.335 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.335 DEBUG:boto:Data: 2023-09-29T22:22:52.335 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.336 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.336 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.336 DEBUG:boto:Params: {} 2023-09-29T22:22:52.336 DEBUG:boto:Token: None 2023-09-29T22:22:52.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E= 2023-09-29T22:22:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E=', 'Content-Length': '0'} 2023-09-29T22:22:52.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f9562fd92d906a8-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.338 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.338 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.338 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.339 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.339 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.339 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.339 DEBUG:boto:Data: 2023-09-29T22:22:52.339 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.339 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.339 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.339 DEBUG:boto:Params: {} 2023-09-29T22:22:52.339 DEBUG:boto:Token: None 2023-09-29T22:22:52.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aNcIo3aJuJIsN3YinqEoQaLm3qQ= 2023-09-29T22:22:52.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aNcIo3aJuJIsN3YinqEoQaLm3qQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057f427e7dd00a1e0-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.341 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.342 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.342 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.342 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.342 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.342 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.342 DEBUG:boto:Data: 2023-09-29T22:22:52.342 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.342 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.342 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.343 DEBUG:boto:Params: {} 2023-09-29T22:22:52.343 DEBUG:boto:Token: None 2023-09-29T22:22:52.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.343 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aNcIo3aJuJIsN3YinqEoQaLm3qQ= 2023-09-29T22:22:52.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aNcIo3aJuJIsN3YinqEoQaLm3qQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fa643f83ada3ea2-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.345 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.345 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.345 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.346 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.346 DEBUG:boto:Method: GET 2023-09-29T22:22:52.346 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.346 DEBUG:boto:Data: 2023-09-29T22:22:52.346 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.346 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.346 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.346 DEBUG:boto:Params: {} 2023-09-29T22:22:52.346 DEBUG:boto:Token: None 2023-09-29T22:22:52.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E= 2023-09-29T22:22:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E=', 'Content-Length': '0'} 2023-09-29T22:22:52.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c0f333ca39ece21-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.349 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.349 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.349 DEBUG:boto:path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.349 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.349 DEBUG:boto:Method: GET 2023-09-29T22:22:52.349 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.349 DEBUG:boto:Data: 2023-09-29T22:22:52.349 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.349 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.350 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.350 DEBUG:boto:Params: {} 2023-09-29T22:22:52.350 DEBUG:boto:Token: None 2023-09-29T22:22:52.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=H62A4zyj3HF6.PqOXDao2iSXg0kywxS 2023-09-29T22:22:52.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E= 2023-09-29T22:22:52.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uwqcFXioBD6EStVBU4peEn7kz+E=', 'Content-Length': '0'} 2023-09-29T22:22:52.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'H62A4zyj3HF6.PqOXDao2iSXg0kywxS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009aa40baea56b29e-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.394 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.394 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.394 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.395 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.395 DEBUG:boto:Method: GET 2023-09-29T22:22:52.395 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.395 DEBUG:boto:Data: 2023-09-29T22:22:52.395 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.395 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.395 DEBUG:boto:Params: {} 2023-09-29T22:22:52.395 DEBUG:boto:Token: None 2023-09-29T22:22:52.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fee9328b048af9-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.398 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.399 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.399 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.399 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.399 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.399 DEBUG:boto:Method: GET 2023-09-29T22:22:52.399 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.399 DEBUG:boto:Data: 2023-09-29T22:22:52.399 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.400 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.400 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.400 DEBUG:boto:Params: {} 2023-09-29T22:22:52.400 DEBUG:boto:Token: None 2023-09-29T22:22:52.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309893341f6ede2c-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.403 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.403 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.403 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.403 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.403 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.403 DEBUG:boto:Data: 2023-09-29T22:22:52.404 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.404 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.404 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.404 DEBUG:boto:Params: {} 2023-09-29T22:22:52.404 DEBUG:boto:Token: None 2023-09-29T22:22:52.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d81d7c590f2aeae4-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.407 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.407 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.408 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.408 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.408 DEBUG:boto:Data: 2023-09-29T22:22:52.408 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.408 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.408 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.408 DEBUG:boto:Params: {} 2023-09-29T22:22:52.408 DEBUG:boto:Token: None 2023-09-29T22:22:52.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.408 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000113a1f3238c3edc5-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.412 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.412 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.412 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.412 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.412 DEBUG:boto:Method: GET 2023-09-29T22:22:52.412 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.413 DEBUG:boto:Data: 2023-09-29T22:22:52.413 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.413 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.413 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.413 DEBUG:boto:Params: {} 2023-09-29T22:22:52.413 DEBUG:boto:Token: None 2023-09-29T22:22:52.413 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.413 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d6e9d237c62e630-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.458 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.459 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.459 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.459 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.459 DEBUG:boto:Method: GET 2023-09-29T22:22:52.459 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.459 DEBUG:boto:Data: 2023-09-29T22:22:52.459 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.459 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.459 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.460 DEBUG:boto:Params: {} 2023-09-29T22:22:52.460 DEBUG:boto:Token: None 2023-09-29T22:22:52.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe0e69935034d5de-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.502 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.502 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.503 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.503 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.503 DEBUG:boto:Method: GET 2023-09-29T22:22:52.503 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.503 DEBUG:boto:Data: 2023-09-29T22:22:52.503 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.503 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.503 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.503 DEBUG:boto:Params: {} 2023-09-29T22:22:52.503 DEBUG:boto:Token: None 2023-09-29T22:22:52.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d56634cafc928f0f-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.506 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.507 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.507 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.507 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.507 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.507 DEBUG:boto:Method: GET 2023-09-29T22:22:52.507 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.507 DEBUG:boto:Data: 2023-09-29T22:22:52.507 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.508 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.508 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.508 DEBUG:boto:Params: {} 2023-09-29T22:22:52.508 DEBUG:boto:Token: None 2023-09-29T22:22:52.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bef0c80a68167a8-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.511 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.511 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.511 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.511 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.512 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.512 DEBUG:boto:Method: GET 2023-09-29T22:22:52.512 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.512 DEBUG:boto:Data: 2023-09-29T22:22:52.512 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.512 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.512 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.512 DEBUG:boto:Params: {} 2023-09-29T22:22:52.512 DEBUG:boto:Token: None 2023-09-29T22:22:52.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U= 2023-09-29T22:22:52.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U=', 'Content-Length': '0'} 2023-09-29T22:22:52.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd852bc7ade97386-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.515 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.515 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.515 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.515 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.515 DEBUG:boto:Method: GET 2023-09-29T22:22:52.516 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.516 DEBUG:boto:Data: 2023-09-29T22:22:52.516 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.516 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.516 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.516 DEBUG:boto:Params: {} 2023-09-29T22:22:52.516 DEBUG:boto:Token: None 2023-09-29T22:22:52.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U= 2023-09-29T22:22:52.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U=', 'Content-Length': '0'} 2023-09-29T22:22:52.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008713e9f9bc7ea4b7-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.562 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.563 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.563 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.563 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.563 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.563 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.563 DEBUG:boto:Data: 2023-09-29T22:22:52.563 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.563 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.563 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.564 DEBUG:boto:Params: {} 2023-09-29T22:22:52.564 DEBUG:boto:Token: None 2023-09-29T22:22:52.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mueFwQvaUq0DjogfnLoOBUPRVbQ= 2023-09-29T22:22:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mueFwQvaUq0DjogfnLoOBUPRVbQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fd107d2896f5968-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.566 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.566 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.566 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.566 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.566 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.566 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.566 DEBUG:boto:Data: 2023-09-29T22:22:52.567 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.567 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.567 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.567 DEBUG:boto:Params: {} 2023-09-29T22:22:52.567 DEBUG:boto:Token: None 2023-09-29T22:22:52.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.567 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mueFwQvaUq0DjogfnLoOBUPRVbQ= 2023-09-29T22:22:52.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mueFwQvaUq0DjogfnLoOBUPRVbQ=', 'Content-Length': '0'} 2023-09-29T22:22:52.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007747108658015cf8-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.569 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.569 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.570 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.570 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.570 DEBUG:boto:Method: GET 2023-09-29T22:22:52.570 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.570 DEBUG:boto:Data: 2023-09-29T22:22:52.570 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.570 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.570 DEBUG:boto:Params: {} 2023-09-29T22:22:52.571 DEBUG:boto:Token: None 2023-09-29T22:22:52.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U= 2023-09-29T22:22:52.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U=', 'Content-Length': '0'} 2023-09-29T22:22:52.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbd9ea414fe10fe7-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.614 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.614 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.614 DEBUG:boto:path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.615 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.615 DEBUG:boto:Method: GET 2023-09-29T22:22:52.615 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.615 DEBUG:boto:Data: 2023-09-29T22:22:52.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.615 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.615 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.615 DEBUG:boto:Params: {} 2023-09-29T22:22:52.615 DEBUG:boto:Token: None 2023-09-29T22:22:52.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh 2023-09-29T22:22:52.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U= 2023-09-29T22:22:52.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nYMEI1hrLiwPFsCrU2tvWUkeF8U=', 'Content-Length': '0'} 2023-09-29T22:22:52.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'pHUoRnAhPF0mOKRAF4HQOhhBLX0yJHh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae685a9fd0a6772-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.658 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.658 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.659 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.659 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.659 DEBUG:boto:Method: GET 2023-09-29T22:22:52.659 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.659 DEBUG:boto:Data: 2023-09-29T22:22:52.659 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.659 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.659 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.659 DEBUG:boto:Params: {} 2023-09-29T22:22:52.659 DEBUG:boto:Token: None 2023-09-29T22:22:52.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baaf5e3bb3ecbe06-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.662 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.663 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.663 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.663 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.663 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.663 DEBUG:boto:Method: GET 2023-09-29T22:22:52.663 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.663 DEBUG:boto:Data: 2023-09-29T22:22:52.664 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.664 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.664 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.664 DEBUG:boto:Params: {} 2023-09-29T22:22:52.664 DEBUG:boto:Token: None 2023-09-29T22:22:52.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b221d0facc0c30-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.667 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.669 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.669 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.669 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.669 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.669 DEBUG:boto:Data: 2023-09-29T22:22:52.669 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.670 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.670 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.670 DEBUG:boto:Params: {} 2023-09-29T22:22:52.670 DEBUG:boto:Token: None 2023-09-29T22:22:52.670 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.670 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e9cf7a30c184ec1-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.673 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.673 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.674 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.674 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.674 DEBUG:boto:Data: 2023-09-29T22:22:52.674 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.674 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.674 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.674 DEBUG:boto:Params: {} 2023-09-29T22:22:52.674 DEBUG:boto:Token: None 2023-09-29T22:22:52.674 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.675 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd90c204df247aac-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.679 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.679 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.679 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.679 DEBUG:boto:Method: GET 2023-09-29T22:22:52.679 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.679 DEBUG:boto:Data: 2023-09-29T22:22:52.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.680 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.680 DEBUG:boto:Params: {} 2023-09-29T22:22:52.680 DEBUG:boto:Token: None 2023-09-29T22:22:52.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab08c8a76f3a738b-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.722 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.722 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.723 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.723 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.723 DEBUG:boto:Method: GET 2023-09-29T22:22:52.723 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.723 DEBUG:boto:Data: 2023-09-29T22:22:52.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.723 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.723 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.723 DEBUG:boto:Params: {} 2023-09-29T22:22:52.724 DEBUG:boto:Token: None 2023-09-29T22:22:52.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c32ff11df441655f-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.766 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.766 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.767 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.767 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.767 DEBUG:boto:Method: GET 2023-09-29T22:22:52.767 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.767 DEBUG:boto:Data: 2023-09-29T22:22:52.767 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.767 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.767 DEBUG:boto:Params: {} 2023-09-29T22:22:52.768 DEBUG:boto:Token: None 2023-09-29T22:22:52.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d2f6954d2c906f-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.771 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.771 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.771 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.771 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.771 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.771 DEBUG:boto:Method: GET 2023-09-29T22:22:52.772 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.772 DEBUG:boto:Data: 2023-09-29T22:22:52.772 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.772 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.772 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.772 DEBUG:boto:Params: {} 2023-09-29T22:22:52.772 DEBUG:boto:Token: None 2023-09-29T22:22:52.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daaecb0a28708156-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.818 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.819 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.819 DEBUG:boto:Method: GET 2023-09-29T22:22:52.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.819 DEBUG:boto:Data: 2023-09-29T22:22:52.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.820 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.820 DEBUG:boto:Params: {} 2023-09-29T22:22:52.820 DEBUG:boto:Token: None 2023-09-29T22:22:52.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc= 2023-09-29T22:22:52.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc=', 'Content-Length': '0'} 2023-09-29T22:22:52.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9a1c442f033cff7-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.822 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.823 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.823 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.823 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.823 DEBUG:boto:Method: GET 2023-09-29T22:22:52.823 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.823 DEBUG:boto:Data: 2023-09-29T22:22:52.823 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.823 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.823 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.823 DEBUG:boto:Params: {} 2023-09-29T22:22:52.824 DEBUG:boto:Token: None 2023-09-29T22:22:52.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc= 2023-09-29T22:22:52.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc=', 'Content-Length': '0'} 2023-09-29T22:22:52.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc9dcd31c5d9616-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.826 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.826 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.826 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.827 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.827 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.827 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.827 DEBUG:boto:Data: 2023-09-29T22:22:52.827 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.827 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.827 DEBUG:boto:Params: {} 2023-09-29T22:22:52.827 DEBUG:boto:Token: None 2023-09-29T22:22:52.828 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PydBmEgr+qQolUaLvbIDosz4NIw= 2023-09-29T22:22:52.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PydBmEgr+qQolUaLvbIDosz4NIw=', 'Content-Length': '0'} 2023-09-29T22:22:52.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a404e93b6cc7517b-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.830 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.830 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.830 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.830 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.830 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.830 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.830 DEBUG:boto:Data: 2023-09-29T22:22:52.830 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.831 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.831 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.831 DEBUG:boto:Params: {} 2023-09-29T22:22:52.831 DEBUG:boto:Token: None 2023-09-29T22:22:52.831 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PydBmEgr+qQolUaLvbIDosz4NIw= 2023-09-29T22:22:52.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PydBmEgr+qQolUaLvbIDosz4NIw=', 'Content-Length': '0'} 2023-09-29T22:22:52.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0758ca8c8f8381b-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.833 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.833 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.834 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.834 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.834 DEBUG:boto:Method: GET 2023-09-29T22:22:52.834 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.834 DEBUG:boto:Data: 2023-09-29T22:22:52.834 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.834 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.834 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.834 DEBUG:boto:Params: {} 2023-09-29T22:22:52.834 DEBUG:boto:Token: None 2023-09-29T22:22:52.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc= 2023-09-29T22:22:52.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc=', 'Content-Length': '0'} 2023-09-29T22:22:52.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000622e15aef66c3ce3-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.837 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.837 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.837 DEBUG:boto:path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.837 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.837 DEBUG:boto:Method: GET 2023-09-29T22:22:52.837 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.837 DEBUG:boto:Data: 2023-09-29T22:22:52.837 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.838 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.838 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.838 DEBUG:boto:Params: {} 2023-09-29T22:22:52.838 DEBUG:boto:Token: None 2023-09-29T22:22:52.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=2MT28my.DVcM5Kd42w8oaZYB1cPftpt 2023-09-29T22:22:52.838 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc= 2023-09-29T22:22:52.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YFZElTPxnF4toJI0TaMgiuXyXzc=', 'Content-Length': '0'} 2023-09-29T22:22:52.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '2MT28my.DVcM5Kd42w8oaZYB1cPftpt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004adfab371945f64b-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.886 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.886 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.887 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.887 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.887 DEBUG:boto:Method: GET 2023-09-29T22:22:52.887 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.887 DEBUG:boto:Data: 2023-09-29T22:22:52.887 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.887 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.887 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.887 DEBUG:boto:Params: {} 2023-09-29T22:22:52.887 DEBUG:boto:Token: None 2023-09-29T22:22:52.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c02a4b17345460-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.890 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.891 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.891 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.891 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.891 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.891 DEBUG:boto:Method: GET 2023-09-29T22:22:52.891 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.891 DEBUG:boto:Data: 2023-09-29T22:22:52.892 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.892 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.892 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.892 DEBUG:boto:Params: {} 2023-09-29T22:22:52.892 DEBUG:boto:Token: None 2023-09-29T22:22:52.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/ 2023-09-29T22:22:52.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU= 2023-09-29T22:22:52.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h4oxntgwMUcIlUcLikhEEgx7QpU=', 'Content-Length': '0'} 2023-09-29T22:22:52.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001307a240683c3d50-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.895 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:52.895 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.896 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.896 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.896 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.896 DEBUG:boto:Data: 2023-09-29T22:22:52.896 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.896 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.896 DEBUG:boto:Params: {} 2023-09-29T22:22:52.896 DEBUG:boto:Token: None 2023-09-29T22:22:52.897 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ed0510501ae624b-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.900 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.900 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.900 DEBUG:boto:Method: HEAD 2023-09-29T22:22:52.900 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:52.900 DEBUG:boto:Data: 2023-09-29T22:22:52.900 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.900 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.900 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.901 DEBUG:boto:Params: {} 2023-09-29T22:22:52.901 DEBUG:boto:Token: None 2023-09-29T22:22:52.901 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj 2023-09-29T22:22:52.901 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8= 2023-09-29T22:22:52.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:neEdBhlyYlkILvkcwdjTBHW/Ho8=', 'Content-Length': '0'} 2023-09-29T22:22:52.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a526a814854712d-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:52.904 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.905 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.905 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.905 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.905 DEBUG:boto:Method: GET 2023-09-29T22:22:52.905 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.905 DEBUG:boto:Data: 2023-09-29T22:22:52.905 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.905 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.905 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.905 DEBUG:boto:Params: {} 2023-09-29T22:22:52.906 DEBUG:boto:Token: None 2023-09-29T22:22:52.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.906 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f507e015dbf5d5d5-0065174e3c-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.950 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:52.951 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:52.951 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.951 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.951 DEBUG:boto:Method: GET 2023-09-29T22:22:52.951 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.951 DEBUG:boto:Data: 2023-09-29T22:22:52.951 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.951 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:52.951 DEBUG:boto:Port: 8000 2023-09-29T22:22:52.952 DEBUG:boto:Params: {} 2023-09-29T22:22:52.952 DEBUG:boto:Token: None 2023-09-29T22:22:52.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:52 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:52.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s= 2023-09-29T22:22:52.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8Mb5J4y2UBvAKaD83E//rnh7i0s=', 'Content-Length': '0'} 2023-09-29T22:22:52.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad517822ca092070-0065174e3c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:52.998 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:52.998 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:52.999 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.999 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.999 DEBUG:boto:Method: GET 2023-09-29T22:22:52.999 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:52.999 DEBUG:boto:Data: 2023-09-29T22:22:52.999 DEBUG:boto:Headers: {} 2023-09-29T22:22:52.999 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:52.999 DEBUG:boto:Port: 8001 2023-09-29T22:22:52.999 DEBUG:boto:Params: {} 2023-09-29T22:22:52.999 DEBUG:boto:Token: None 2023-09-29T22:22:53.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa60a5ba54c22a5e-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.002 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.003 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.003 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.003 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.003 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.003 DEBUG:boto:Method: GET 2023-09-29T22:22:53.003 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.004 DEBUG:boto:Data: 2023-09-29T22:22:53.004 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.004 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.004 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.004 DEBUG:boto:Params: {} 2023-09-29T22:22:53.004 DEBUG:boto:Token: None 2023-09-29T22:22:53.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da72adda54208138-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.007 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.008 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.008 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.008 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.008 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.008 DEBUG:boto:Method: GET 2023-09-29T22:22:53.008 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.008 DEBUG:boto:Data: 2023-09-29T22:22:53.008 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.008 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.008 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.009 DEBUG:boto:Params: {} 2023-09-29T22:22:53.009 DEBUG:boto:Token: None 2023-09-29T22:22:53.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.009 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg= 2023-09-29T22:22:53.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg=', 'Content-Length': '0'} 2023-09-29T22:22:53.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e8071cb8b74833-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.011 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.011 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.011 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.012 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.012 DEBUG:boto:Method: GET 2023-09-29T22:22:53.012 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.012 DEBUG:boto:Data: 2023-09-29T22:22:53.012 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.012 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.012 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.012 DEBUG:boto:Params: {} 2023-09-29T22:22:53.012 DEBUG:boto:Token: None 2023-09-29T22:22:53.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.013 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg= 2023-09-29T22:22:53.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg=', 'Content-Length': '0'} 2023-09-29T22:22:53.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce8126cbc832ff47-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.062 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.063 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.063 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.063 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.063 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.063 DEBUG:boto:Data: 2023-09-29T22:22:53.063 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.063 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.064 DEBUG:boto:Params: {} 2023-09-29T22:22:53.064 DEBUG:boto:Token: None 2023-09-29T22:22:53.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8U2ZFXyzyHVSzSitQbjZEVxWK28= 2023-09-29T22:22:53.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8U2ZFXyzyHVSzSitQbjZEVxWK28=', 'Content-Length': '0'} 2023-09-29T22:22:53.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d10344c617269a1d-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.066 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.066 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.066 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.066 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.067 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.067 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.067 DEBUG:boto:Data: 2023-09-29T22:22:53.067 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.067 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.067 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.067 DEBUG:boto:Params: {} 2023-09-29T22:22:53.067 DEBUG:boto:Token: None 2023-09-29T22:22:53.067 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8U2ZFXyzyHVSzSitQbjZEVxWK28= 2023-09-29T22:22:53.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8U2ZFXyzyHVSzSitQbjZEVxWK28=', 'Content-Length': '0'} 2023-09-29T22:22:53.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e73f820057f7051-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.070 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.070 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.070 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.070 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.070 DEBUG:boto:Method: GET 2023-09-29T22:22:53.070 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.070 DEBUG:boto:Data: 2023-09-29T22:22:53.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.071 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.071 DEBUG:boto:Params: {} 2023-09-29T22:22:53.071 DEBUG:boto:Token: None 2023-09-29T22:22:53.071 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.071 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg= 2023-09-29T22:22:53.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg=', 'Content-Length': '0'} 2023-09-29T22:22:53.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaec964ff952263b-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.114 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.115 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.115 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.115 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.115 DEBUG:boto:Method: GET 2023-09-29T22:22:53.115 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.115 DEBUG:boto:Data: 2023-09-29T22:22:53.115 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.115 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.115 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.115 DEBUG:boto:Params: {} 2023-09-29T22:22:53.116 DEBUG:boto:Token: None 2023-09-29T22:22:53.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ARt2wNmFlEw7PolmrP0bT3B51j6z7ox 2023-09-29T22:22:53.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg= 2023-09-29T22:22:53.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FfdjlXnawonR6JFnd7Tl/ZzxCyg=', 'Content-Length': '0'} 2023-09-29T22:22:53.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ARt2wNmFlEw7PolmrP0bT3B51j6z7ox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007067270146c8c679-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.162 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.163 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.163 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.163 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.163 DEBUG:boto:Method: GET 2023-09-29T22:22:53.163 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.163 DEBUG:boto:Data: 2023-09-29T22:22:53.163 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.163 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.164 DEBUG:boto:Params: {} 2023-09-29T22:22:53.164 DEBUG:boto:Token: None 2023-09-29T22:22:53.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.164 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f604a925ffeb2578-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.167 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.167 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.168 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.168 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.168 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.168 DEBUG:boto:Method: GET 2023-09-29T22:22:53.168 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.168 DEBUG:boto:Data: 2023-09-29T22:22:53.168 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.168 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.168 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.168 DEBUG:boto:Params: {} 2023-09-29T22:22:53.168 DEBUG:boto:Token: None 2023-09-29T22:22:53.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c0ca892d5ecb462-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.171 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.172 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.172 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.172 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.172 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.172 DEBUG:boto:Data: 2023-09-29T22:22:53.172 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.173 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.173 DEBUG:boto:Params: {} 2023-09-29T22:22:53.173 DEBUG:boto:Token: None 2023-09-29T22:22:53.173 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.173 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b8bca26bedb24a6-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.176 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.177 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.177 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.177 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.177 DEBUG:boto:Data: 2023-09-29T22:22:53.177 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.177 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.177 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.177 DEBUG:boto:Params: {} 2023-09-29T22:22:53.177 DEBUG:boto:Token: None 2023-09-29T22:22:53.178 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.178 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000033462fa3bcacf33-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.181 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.181 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.181 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.182 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.182 DEBUG:boto:Method: GET 2023-09-29T22:22:53.182 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.182 DEBUG:boto:Data: 2023-09-29T22:22:53.182 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.182 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.182 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.182 DEBUG:boto:Params: {} 2023-09-29T22:22:53.182 DEBUG:boto:Token: None 2023-09-29T22:22:53.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cd847c2233db55f-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.226 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.226 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.227 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.227 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.227 DEBUG:boto:Method: GET 2023-09-29T22:22:53.227 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.227 DEBUG:boto:Data: 2023-09-29T22:22:53.227 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.227 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.227 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.227 DEBUG:boto:Params: {} 2023-09-29T22:22:53.227 DEBUG:boto:Token: None 2023-09-29T22:22:53.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000396a009e56f60781-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.270 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.270 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.271 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.271 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.271 DEBUG:boto:Method: GET 2023-09-29T22:22:53.271 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.271 DEBUG:boto:Data: 2023-09-29T22:22:53.271 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.271 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.271 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.271 DEBUG:boto:Params: {} 2023-09-29T22:22:53.271 DEBUG:boto:Token: None 2023-09-29T22:22:53.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25d12ea0f98eff7-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.275 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.275 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.275 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.275 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.275 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.275 DEBUG:boto:Method: GET 2023-09-29T22:22:53.276 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.276 DEBUG:boto:Data: 2023-09-29T22:22:53.276 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.276 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.276 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.276 DEBUG:boto:Params: {} 2023-09-29T22:22:53.276 DEBUG:boto:Token: None 2023-09-29T22:22:53.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52221ee533da16d-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.322 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.323 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.323 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.323 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.323 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.323 DEBUG:boto:Method: GET 2023-09-29T22:22:53.323 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.323 DEBUG:boto:Data: 2023-09-29T22:22:53.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.323 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.324 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.324 DEBUG:boto:Params: {} 2023-09-29T22:22:53.324 DEBUG:boto:Token: None 2023-09-29T22:22:53.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg= 2023-09-29T22:22:53.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg=', 'Content-Length': '0'} 2023-09-29T22:22:53.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005421ee91aee4c773-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.326 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.326 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.326 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.327 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.327 DEBUG:boto:Method: GET 2023-09-29T22:22:53.327 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.327 DEBUG:boto:Data: 2023-09-29T22:22:53.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.327 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.327 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.327 DEBUG:boto:Params: {} 2023-09-29T22:22:53.327 DEBUG:boto:Token: None 2023-09-29T22:22:53.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg= 2023-09-29T22:22:53.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg=', 'Content-Length': '0'} 2023-09-29T22:22:53.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f0210d0656fe08-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.330 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.330 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.330 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.330 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.330 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.330 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.330 DEBUG:boto:Data: 2023-09-29T22:22:53.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.331 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.331 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.331 DEBUG:boto:Params: {} 2023-09-29T22:22:53.331 DEBUG:boto:Token: None 2023-09-29T22:22:53.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jpiUgF123xxDIi8itMO/lXLkVIs= 2023-09-29T22:22:53.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jpiUgF123xxDIi8itMO/lXLkVIs=', 'Content-Length': '0'} 2023-09-29T22:22:53.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000352f9ee4be13d70c-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.333 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.333 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.333 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.333 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.334 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.334 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.334 DEBUG:boto:Data: 2023-09-29T22:22:53.334 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.334 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.334 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.334 DEBUG:boto:Params: {} 2023-09-29T22:22:53.334 DEBUG:boto:Token: None 2023-09-29T22:22:53.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.334 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jpiUgF123xxDIi8itMO/lXLkVIs= 2023-09-29T22:22:53.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jpiUgF123xxDIi8itMO/lXLkVIs=', 'Content-Length': '0'} 2023-09-29T22:22:53.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed712c438c611bb3-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.337 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.337 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.337 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.337 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.337 DEBUG:boto:Method: GET 2023-09-29T22:22:53.337 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.337 DEBUG:boto:Data: 2023-09-29T22:22:53.337 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.337 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.338 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.338 DEBUG:boto:Params: {} 2023-09-29T22:22:53.338 DEBUG:boto:Token: None 2023-09-29T22:22:53.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg= 2023-09-29T22:22:53.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg=', 'Content-Length': '0'} 2023-09-29T22:22:53.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc7aad686235fbf0-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.382 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.383 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.383 DEBUG:boto:path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.383 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.383 DEBUG:boto:Method: GET 2023-09-29T22:22:53.383 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.383 DEBUG:boto:Data: 2023-09-29T22:22:53.383 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.383 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.383 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.383 DEBUG:boto:Params: {} 2023-09-29T22:22:53.383 DEBUG:boto:Token: None 2023-09-29T22:22:53.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=dbJpbtOestfBMgpk.e7uPehxzJXPIMH 2023-09-29T22:22:53.384 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg= 2023-09-29T22:22:53.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3jiny7PLBMjvP/HbY9Gi2JyuXEg=', 'Content-Length': '0'} 2023-09-29T22:22:53.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'dbJpbtOestfBMgpk.e7uPehxzJXPIMH'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099d14f839233afd4-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.386 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.386 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.386 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.387 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.387 DEBUG:boto:Method: GET 2023-09-29T22:22:53.387 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.387 DEBUG:boto:Data: 2023-09-29T22:22:53.387 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.387 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.387 DEBUG:boto:Params: {} 2023-09-29T22:22:53.387 DEBUG:boto:Token: None 2023-09-29T22:22:53.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008304b51548086900-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.390 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.391 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.391 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.391 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.391 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.391 DEBUG:boto:Method: GET 2023-09-29T22:22:53.391 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.391 DEBUG:boto:Data: 2023-09-29T22:22:53.391 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.392 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.392 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.392 DEBUG:boto:Params: {} 2023-09-29T22:22:53.392 DEBUG:boto:Token: None 2023-09-29T22:22:53.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194019d219be4840-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.395 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.395 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.395 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.395 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.396 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.396 DEBUG:boto:Data: 2023-09-29T22:22:53.396 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.396 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.396 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.396 DEBUG:boto:Params: {} 2023-09-29T22:22:53.396 DEBUG:boto:Token: None 2023-09-29T22:22:53.396 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8637fde5734ee34-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.400 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.400 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.400 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.400 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.400 DEBUG:boto:Data: 2023-09-29T22:22:53.400 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.400 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.400 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.400 DEBUG:boto:Params: {} 2023-09-29T22:22:53.401 DEBUG:boto:Token: None 2023-09-29T22:22:53.401 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.401 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006592882658087d69-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.404 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.404 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.404 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.404 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.404 DEBUG:boto:Method: GET 2023-09-29T22:22:53.405 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.405 DEBUG:boto:Data: 2023-09-29T22:22:53.405 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.405 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.405 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.405 DEBUG:boto:Params: {} 2023-09-29T22:22:53.405 DEBUG:boto:Token: None 2023-09-29T22:22:53.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.405 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f113899f2f15ee60-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.450 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.451 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.451 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.451 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.451 DEBUG:boto:Method: GET 2023-09-29T22:22:53.451 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.451 DEBUG:boto:Data: 2023-09-29T22:22:53.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.451 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.451 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.451 DEBUG:boto:Params: {} 2023-09-29T22:22:53.452 DEBUG:boto:Token: None 2023-09-29T22:22:53.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000101d3cbd9e88d974-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.494 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.494 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.495 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.495 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.495 DEBUG:boto:Method: GET 2023-09-29T22:22:53.495 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.495 DEBUG:boto:Data: 2023-09-29T22:22:53.495 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.495 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.495 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.495 DEBUG:boto:Params: {} 2023-09-29T22:22:53.496 DEBUG:boto:Token: None 2023-09-29T22:22:53.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000606247e0a7325380-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.499 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.499 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.499 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.499 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.499 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.499 DEBUG:boto:Method: GET 2023-09-29T22:22:53.500 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.500 DEBUG:boto:Data: 2023-09-29T22:22:53.500 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.500 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.500 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.500 DEBUG:boto:Params: {} 2023-09-29T22:22:53.500 DEBUG:boto:Token: None 2023-09-29T22:22:53.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000885f2aec9e7bc549-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.503 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.504 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.504 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.504 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.504 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.504 DEBUG:boto:Method: GET 2023-09-29T22:22:53.504 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.504 DEBUG:boto:Data: 2023-09-29T22:22:53.504 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.504 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.504 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.504 DEBUG:boto:Params: {} 2023-09-29T22:22:53.505 DEBUG:boto:Token: None 2023-09-29T22:22:53.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0= 2023-09-29T22:22:53.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0=', 'Content-Length': '0'} 2023-09-29T22:22:53.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f6c0ef5c0e444ee-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.507 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.507 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.507 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.508 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.508 DEBUG:boto:Method: GET 2023-09-29T22:22:53.508 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.508 DEBUG:boto:Data: 2023-09-29T22:22:53.508 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.508 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.508 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.508 DEBUG:boto:Params: {} 2023-09-29T22:22:53.508 DEBUG:boto:Token: None 2023-09-29T22:22:53.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0= 2023-09-29T22:22:53.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0=', 'Content-Length': '0'} 2023-09-29T22:22:53.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dbad03d1090bef5-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.511 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.511 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.511 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.511 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.511 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.511 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.511 DEBUG:boto:Data: 2023-09-29T22:22:53.512 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.512 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.512 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.512 DEBUG:boto:Params: {} 2023-09-29T22:22:53.512 DEBUG:boto:Token: None 2023-09-29T22:22:53.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.512 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoKGmJMU+V/BHECnKudiSpufas8= 2023-09-29T22:22:53.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoKGmJMU+V/BHECnKudiSpufas8=', 'Content-Length': '0'} 2023-09-29T22:22:53.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6cdc51e035c0cf5-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.514 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.514 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.514 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.514 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.515 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.515 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.515 DEBUG:boto:Data: 2023-09-29T22:22:53.515 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.515 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.515 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.515 DEBUG:boto:Params: {} 2023-09-29T22:22:53.515 DEBUG:boto:Token: None 2023-09-29T22:22:53.515 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LoKGmJMU+V/BHECnKudiSpufas8= 2023-09-29T22:22:53.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LoKGmJMU+V/BHECnKudiSpufas8=', 'Content-Length': '0'} 2023-09-29T22:22:53.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6696c568ced52fb-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.518 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.518 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.518 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.518 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.518 DEBUG:boto:Method: GET 2023-09-29T22:22:53.518 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.519 DEBUG:boto:Data: 2023-09-29T22:22:53.519 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.519 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.519 DEBUG:boto:Params: {} 2023-09-29T22:22:53.519 DEBUG:boto:Token: None 2023-09-29T22:22:53.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.519 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0= 2023-09-29T22:22:53.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0=', 'Content-Length': '0'} 2023-09-29T22:22:53.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b61fcba96a6b637f-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.562 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.563 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.563 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.563 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.563 DEBUG:boto:Method: GET 2023-09-29T22:22:53.563 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.563 DEBUG:boto:Data: 2023-09-29T22:22:53.563 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.563 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.563 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.563 DEBUG:boto:Params: {} 2023-09-29T22:22:53.564 DEBUG:boto:Token: None 2023-09-29T22:22:53.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw 2023-09-29T22:22:53.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0= 2023-09-29T22:22:53.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ylbbom2tCyKBem4+HPSM2q/1Ak0=', 'Content-Length': '0'} 2023-09-29T22:22:53.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hBI5xQc0xpClMdbPsqNZ0ltzLt4HItw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc845d14d9e80957-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.566 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.566 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.566 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.567 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.567 DEBUG:boto:Method: GET 2023-09-29T22:22:53.567 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.567 DEBUG:boto:Data: 2023-09-29T22:22:53.567 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.567 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.567 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.567 DEBUG:boto:Params: {} 2023-09-29T22:22:53.567 DEBUG:boto:Token: None 2023-09-29T22:22:53.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.568 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1aa284a6dd18f0e-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.614 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.615 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.615 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.615 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.615 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.616 DEBUG:boto:Method: GET 2023-09-29T22:22:53.616 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.616 DEBUG:boto:Data: 2023-09-29T22:22:53.616 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.616 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.616 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.616 DEBUG:boto:Params: {} 2023-09-29T22:22:53.616 DEBUG:boto:Token: None 2023-09-29T22:22:53.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.617 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ed88ea2fbcab19-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.619 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.620 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.620 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.620 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.620 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.620 DEBUG:boto:Data: 2023-09-29T22:22:53.620 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.620 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.621 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.621 DEBUG:boto:Params: {} 2023-09-29T22:22:53.621 DEBUG:boto:Token: None 2023-09-29T22:22:53.621 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.621 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f33e7bcdc96dff30-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.624 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.624 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.624 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.624 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.625 DEBUG:boto:Data: 2023-09-29T22:22:53.625 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.625 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.625 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.625 DEBUG:boto:Params: {} 2023-09-29T22:22:53.625 DEBUG:boto:Token: None 2023-09-29T22:22:53.625 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f65784ddfe70f2d-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.629 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.629 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.629 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.629 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.629 DEBUG:boto:Method: GET 2023-09-29T22:22:53.629 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.629 DEBUG:boto:Data: 2023-09-29T22:22:53.629 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.629 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.630 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.630 DEBUG:boto:Params: {} 2023-09-29T22:22:53.630 DEBUG:boto:Token: None 2023-09-29T22:22:53.630 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.630 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021bc08c078d56d65-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.678 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.679 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.679 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.679 DEBUG:boto:Method: GET 2023-09-29T22:22:53.679 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.679 DEBUG:boto:Data: 2023-09-29T22:22:53.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.679 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.679 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.680 DEBUG:boto:Params: {} 2023-09-29T22:22:53.680 DEBUG:boto:Token: None 2023-09-29T22:22:53.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee129d27d7d0abe6-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.722 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.723 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.723 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.723 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.723 DEBUG:boto:Method: GET 2023-09-29T22:22:53.723 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.723 DEBUG:boto:Data: 2023-09-29T22:22:53.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.723 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.724 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.724 DEBUG:boto:Params: {} 2023-09-29T22:22:53.724 DEBUG:boto:Token: None 2023-09-29T22:22:53.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84ea787467a3208-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.770 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.771 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.771 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.771 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.771 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.771 DEBUG:boto:Method: GET 2023-09-29T22:22:53.771 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.772 DEBUG:boto:Data: 2023-09-29T22:22:53.772 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.772 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.772 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.772 DEBUG:boto:Params: {} 2023-09-29T22:22:53.772 DEBUG:boto:Token: None 2023-09-29T22:22:53.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd36d3a7c02af5d6-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.818 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.819 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.819 DEBUG:boto:Method: GET 2023-09-29T22:22:53.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.820 DEBUG:boto:Data: 2023-09-29T22:22:53.820 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.820 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.820 DEBUG:boto:Params: {} 2023-09-29T22:22:53.820 DEBUG:boto:Token: None 2023-09-29T22:22:53.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU= 2023-09-29T22:22:53.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU=', 'Content-Length': '0'} 2023-09-29T22:22:53.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025afef8b6c704ac7-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.822 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.823 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.823 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.823 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.823 DEBUG:boto:Method: GET 2023-09-29T22:22:53.823 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.823 DEBUG:boto:Data: 2023-09-29T22:22:53.823 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.823 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.823 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.823 DEBUG:boto:Params: {} 2023-09-29T22:22:53.824 DEBUG:boto:Token: None 2023-09-29T22:22:53.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU= 2023-09-29T22:22:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU=', 'Content-Length': '0'} 2023-09-29T22:22:53.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032cd7798d39894a2-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.826 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.826 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.826 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.826 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.827 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.827 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.827 DEBUG:boto:Data: 2023-09-29T22:22:53.827 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.827 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.827 DEBUG:boto:Params: {} 2023-09-29T22:22:53.827 DEBUG:boto:Token: None 2023-09-29T22:22:53.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lIGBazyH5v0dyIcjGYlEXFxTunU= 2023-09-29T22:22:53.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lIGBazyH5v0dyIcjGYlEXFxTunU=', 'Content-Length': '0'} 2023-09-29T22:22:53.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000510901cf88890e32-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.829 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.830 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.830 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.830 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.830 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.830 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.830 DEBUG:boto:Data: 2023-09-29T22:22:53.830 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.830 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.830 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.830 DEBUG:boto:Params: {} 2023-09-29T22:22:53.831 DEBUG:boto:Token: None 2023-09-29T22:22:53.831 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lIGBazyH5v0dyIcjGYlEXFxTunU= 2023-09-29T22:22:53.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lIGBazyH5v0dyIcjGYlEXFxTunU=', 'Content-Length': '0'} 2023-09-29T22:22:53.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0a64bc4c8daca65-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.833 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.833 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.833 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.834 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.834 DEBUG:boto:Method: GET 2023-09-29T22:22:53.834 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.834 DEBUG:boto:Data: 2023-09-29T22:22:53.834 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.834 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.834 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.834 DEBUG:boto:Params: {} 2023-09-29T22:22:53.834 DEBUG:boto:Token: None 2023-09-29T22:22:53.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU= 2023-09-29T22:22:53.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU=', 'Content-Length': '0'} 2023-09-29T22:22:53.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000558b09e8e04476b1-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.837 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.837 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.837 DEBUG:boto:path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.837 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.837 DEBUG:boto:Method: GET 2023-09-29T22:22:53.837 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.838 DEBUG:boto:Data: 2023-09-29T22:22:53.838 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.838 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.838 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.838 DEBUG:boto:Params: {} 2023-09-29T22:22:53.838 DEBUG:boto:Token: None 2023-09-29T22:22:53.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd 2023-09-29T22:22:53.838 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU= 2023-09-29T22:22:53.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vtd0hOClho+i/P4rRXo8faIo1zU=', 'Content-Length': '0'} 2023-09-29T22:22:53.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'ut4FW9RW-2lP6ExrM4RvjthXiEpdQwd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d50b326e50276a5-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.882 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.882 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.883 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.883 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.883 DEBUG:boto:Method: GET 2023-09-29T22:22:53.883 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.883 DEBUG:boto:Data: 2023-09-29T22:22:53.883 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.883 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.883 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.883 DEBUG:boto:Params: {} 2023-09-29T22:22:53.883 DEBUG:boto:Token: None 2023-09-29T22:22:53.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097815cc8fde2140c-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.886 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.887 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.887 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.887 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.887 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.887 DEBUG:boto:Method: GET 2023-09-29T22:22:53.887 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.887 DEBUG:boto:Data: 2023-09-29T22:22:53.887 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.888 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.888 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.888 DEBUG:boto:Params: {} 2023-09-29T22:22:53.888 DEBUG:boto:Token: None 2023-09-29T22:22:53.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae6f46495d24addc-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.891 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.893 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.893 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.893 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.893 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.893 DEBUG:boto:Data: 2023-09-29T22:22:53.893 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.893 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.894 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.894 DEBUG:boto:Params: {} 2023-09-29T22:22:53.894 DEBUG:boto:Token: None 2023-09-29T22:22:53.894 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.894 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bb556496d944478-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.897 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.897 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.897 DEBUG:boto:Method: HEAD 2023-09-29T22:22:53.897 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:53.898 DEBUG:boto:Data: 2023-09-29T22:22:53.898 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.898 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.898 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.898 DEBUG:boto:Params: {} 2023-09-29T22:22:53.898 DEBUG:boto:Token: None 2023-09-29T22:22:53.898 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj 2023-09-29T22:22:53.898 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw= 2023-09-29T22:22:53.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UcQSjdDReGUg/FdhPpdRg6PEhLw=', 'Content-Length': '0'} 2023-09-29T22:22:53.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000366676bfa8634e99-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:53.902 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.902 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.902 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.902 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.902 DEBUG:boto:Method: GET 2023-09-29T22:22:53.902 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.902 DEBUG:boto:Data: 2023-09-29T22:22:53.903 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.903 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.903 DEBUG:boto:Params: {} 2023-09-29T22:22:53.903 DEBUG:boto:Token: None 2023-09-29T22:22:53.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.903 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056aa82cece3b1c54-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.946 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:53.946 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:53.946 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.947 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.947 DEBUG:boto:Method: GET 2023-09-29T22:22:53.947 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.947 DEBUG:boto:Data: 2023-09-29T22:22:53.947 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.947 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.947 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.947 DEBUG:boto:Params: {} 2023-09-29T22:22:53.947 DEBUG:boto:Token: None 2023-09-29T22:22:53.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:53.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc= 2023-09-29T22:22:53.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:64Mo0gywEtS5A/VA/Ttzbm+nRIc=', 'Content-Length': '0'} 2023-09-29T22:22:53.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e2c245d0f5f1e99-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.990 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.991 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.991 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.991 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.991 DEBUG:boto:Method: GET 2023-09-29T22:22:53.991 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.991 DEBUG:boto:Data: 2023-09-29T22:22:53.991 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.991 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:53.991 DEBUG:boto:Port: 8001 2023-09-29T22:22:53.992 DEBUG:boto:Params: {} 2023-09-29T22:22:53.992 DEBUG:boto:Token: None 2023-09-29T22:22:53.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008623c88115f44d6c-0065174e3d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.995 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.995 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:53.995 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:53.995 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.995 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.995 DEBUG:boto:Method: GET 2023-09-29T22:22:53.995 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:53.996 DEBUG:boto:Data: 2023-09-29T22:22:53.996 DEBUG:boto:Headers: {} 2023-09-29T22:22:53.996 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:53.996 DEBUG:boto:Port: 8000 2023-09-29T22:22:53.996 DEBUG:boto:Params: {} 2023-09-29T22:22:53.996 DEBUG:boto:Token: None 2023-09-29T22:22:53.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:53 GMT /dmuhsu-19/ 2023-09-29T22:22:53.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU= 2023-09-29T22:22:53.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9zMA+Vep6oCPEaeRzUY9SWe4odU=', 'Content-Length': '0'} 2023-09-29T22:22:53.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e580ad8b300408b9-0065174e3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:53.999 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:53.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.000 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.000 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.000 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.000 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.000 DEBUG:boto:Method: GET 2023-09-29T22:22:54.000 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.000 DEBUG:boto:Data: 2023-09-29T22:22:54.000 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.001 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.001 DEBUG:boto:Params: {} 2023-09-29T22:22:54.001 DEBUG:boto:Token: None 2023-09-29T22:22:54.001 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.001 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA= 2023-09-29T22:22:54.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA=', 'Content-Length': '0'} 2023-09-29T22:22:54.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3d6fd10dfa8541b-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.003 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.003 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.003 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.003 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.004 DEBUG:boto:Method: GET 2023-09-29T22:22:54.004 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.004 DEBUG:boto:Data: 2023-09-29T22:22:54.004 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.004 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.004 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.004 DEBUG:boto:Params: {} 2023-09-29T22:22:54.004 DEBUG:boto:Token: None 2023-09-29T22:22:54.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA= 2023-09-29T22:22:54.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA=', 'Content-Length': '0'} 2023-09-29T22:22:54.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd7c7e153a84e664-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.050 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.051 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.051 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.051 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.051 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.051 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.051 DEBUG:boto:Data: 2023-09-29T22:22:54.051 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.051 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.051 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.052 DEBUG:boto:Params: {} 2023-09-29T22:22:54.052 DEBUG:boto:Token: None 2023-09-29T22:22:54.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VJBUpUtIMl0eV7x6WdusezBQGXs= 2023-09-29T22:22:54.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VJBUpUtIMl0eV7x6WdusezBQGXs=', 'Content-Length': '0'} 2023-09-29T22:22:54.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710d6434882790e1-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.054 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.055 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.055 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.055 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.055 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.055 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.055 DEBUG:boto:Data: 2023-09-29T22:22:54.055 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.055 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.055 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.056 DEBUG:boto:Params: {} 2023-09-29T22:22:54.056 DEBUG:boto:Token: None 2023-09-29T22:22:54.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VJBUpUtIMl0eV7x6WdusezBQGXs= 2023-09-29T22:22:54.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VJBUpUtIMl0eV7x6WdusezBQGXs=', 'Content-Length': '0'} 2023-09-29T22:22:54.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43af78ea5bd36ee-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.058 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.058 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.058 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.059 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.059 DEBUG:boto:Method: GET 2023-09-29T22:22:54.059 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.059 DEBUG:boto:Data: 2023-09-29T22:22:54.059 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.059 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.059 DEBUG:boto:Params: {} 2023-09-29T22:22:54.059 DEBUG:boto:Token: None 2023-09-29T22:22:54.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA= 2023-09-29T22:22:54.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA=', 'Content-Length': '0'} 2023-09-29T22:22:54.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001682946253ada4b6-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.107 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.107 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.108 DEBUG:boto:path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.108 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.108 DEBUG:boto:Method: GET 2023-09-29T22:22:54.108 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.108 DEBUG:boto:Data: 2023-09-29T22:22:54.108 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.108 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.108 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.108 DEBUG:boto:Params: {} 2023-09-29T22:22:54.109 DEBUG:boto:Token: None 2023-09-29T22:22:54.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ 2023-09-29T22:22:54.109 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA= 2023-09-29T22:22:54.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1gKvmYKehsu20Fc22gLsx+9j5mA=', 'Content-Length': '0'} 2023-09-29T22:22:54.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'xaaQJ046Z6VF8k3Ro1ny.a1wQi3lpJJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea8a4f2a9281647f-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.158 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.159 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.159 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.159 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.159 DEBUG:boto:Method: GET 2023-09-29T22:22:54.159 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.159 DEBUG:boto:Data: 2023-09-29T22:22:54.159 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.159 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.159 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.160 DEBUG:boto:Params: {} 2023-09-29T22:22:54.160 DEBUG:boto:Token: None 2023-09-29T22:22:54.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.160 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eea3fe84e6ce141-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.163 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.164 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.164 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.164 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.164 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.164 DEBUG:boto:Method: GET 2023-09-29T22:22:54.164 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.164 DEBUG:boto:Data: 2023-09-29T22:22:54.164 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.165 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.165 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.165 DEBUG:boto:Params: {} 2023-09-29T22:22:54.165 DEBUG:boto:Token: None 2023-09-29T22:22:54.165 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692a43ca86e9ec26-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.168 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.168 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.169 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.169 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.169 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.169 DEBUG:boto:Data: 2023-09-29T22:22:54.169 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.169 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.169 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.169 DEBUG:boto:Params: {} 2023-09-29T22:22:54.169 DEBUG:boto:Token: None 2023-09-29T22:22:54.170 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.170 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bec94dbc28fabee0-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.173 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.173 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.173 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.173 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.174 DEBUG:boto:Data: 2023-09-29T22:22:54.174 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.174 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.174 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.174 DEBUG:boto:Params: {} 2023-09-29T22:22:54.174 DEBUG:boto:Token: None 2023-09-29T22:22:54.174 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.174 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036e48d09523d5cd9-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.178 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.178 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.178 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.178 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.178 DEBUG:boto:Method: GET 2023-09-29T22:22:54.178 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.179 DEBUG:boto:Data: 2023-09-29T22:22:54.179 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.179 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.179 DEBUG:boto:Params: {} 2023-09-29T22:22:54.179 DEBUG:boto:Token: None 2023-09-29T22:22:54.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca4bfcc15c51746d-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.222 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.223 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.223 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.223 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.223 DEBUG:boto:Method: GET 2023-09-29T22:22:54.223 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.223 DEBUG:boto:Data: 2023-09-29T22:22:54.224 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.224 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.224 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.224 DEBUG:boto:Params: {} 2023-09-29T22:22:54.224 DEBUG:boto:Token: None 2023-09-29T22:22:54.225 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.225 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f23e780cfd81255f-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.274 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.275 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.275 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.275 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.276 DEBUG:boto:Method: GET 2023-09-29T22:22:54.276 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.276 DEBUG:boto:Data: 2023-09-29T22:22:54.276 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.276 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.276 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.277 DEBUG:boto:Params: {} 2023-09-29T22:22:54.277 DEBUG:boto:Token: None 2023-09-29T22:22:54.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.277 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000827afb27a614214c-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.281 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.281 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.281 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.282 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.282 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.282 DEBUG:boto:Method: GET 2023-09-29T22:22:54.282 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.282 DEBUG:boto:Data: 2023-09-29T22:22:54.282 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.283 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.283 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.283 DEBUG:boto:Params: {} 2023-09-29T22:22:54.283 DEBUG:boto:Token: None 2023-09-29T22:22:54.283 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f68c10eef133fac-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.330 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.331 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.331 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.331 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.331 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.331 DEBUG:boto:Method: GET 2023-09-29T22:22:54.331 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.331 DEBUG:boto:Data: 2023-09-29T22:22:54.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.332 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.332 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.332 DEBUG:boto:Params: {} 2023-09-29T22:22:54.332 DEBUG:boto:Token: None 2023-09-29T22:22:54.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.332 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI= 2023-09-29T22:22:54.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI=', 'Content-Length': '0'} 2023-09-29T22:22:54.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a44c7f9ca36cae74-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.334 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.335 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.335 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.335 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.335 DEBUG:boto:Method: GET 2023-09-29T22:22:54.335 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.335 DEBUG:boto:Data: 2023-09-29T22:22:54.335 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.335 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.335 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.335 DEBUG:boto:Params: {} 2023-09-29T22:22:54.336 DEBUG:boto:Token: None 2023-09-29T22:22:54.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI= 2023-09-29T22:22:54.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI=', 'Content-Length': '0'} 2023-09-29T22:22:54.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd48d04799a42e39-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.338 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.338 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.338 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.338 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.338 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.338 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.338 DEBUG:boto:Data: 2023-09-29T22:22:54.339 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.339 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.339 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.339 DEBUG:boto:Params: {} 2023-09-29T22:22:54.339 DEBUG:boto:Token: None 2023-09-29T22:22:54.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.339 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0BqguyaAeoWIPoo3W/gemT0w8ug= 2023-09-29T22:22:54.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0BqguyaAeoWIPoo3W/gemT0w8ug=', 'Content-Length': '0'} 2023-09-29T22:22:54.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de863f2885236ff2-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.341 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.341 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.341 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.342 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.342 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.342 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.342 DEBUG:boto:Data: 2023-09-29T22:22:54.342 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.343 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.343 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.343 DEBUG:boto:Params: {} 2023-09-29T22:22:54.343 DEBUG:boto:Token: None 2023-09-29T22:22:54.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0BqguyaAeoWIPoo3W/gemT0w8ug= 2023-09-29T22:22:54.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0BqguyaAeoWIPoo3W/gemT0w8ug=', 'Content-Length': '0'} 2023-09-29T22:22:54.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073502680d8e0e2d0-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.346 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.347 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.347 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.347 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.347 DEBUG:boto:Method: GET 2023-09-29T22:22:54.347 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.347 DEBUG:boto:Data: 2023-09-29T22:22:54.348 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.348 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.348 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.348 DEBUG:boto:Params: {} 2023-09-29T22:22:54.348 DEBUG:boto:Token: None 2023-09-29T22:22:54.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.349 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI= 2023-09-29T22:22:54.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI=', 'Content-Length': '0'} 2023-09-29T22:22:54.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000173d0765726d6d53-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.394 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.395 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.395 DEBUG:boto:path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.395 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.395 DEBUG:boto:Method: GET 2023-09-29T22:22:54.395 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.395 DEBUG:boto:Data: 2023-09-29T22:22:54.396 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.396 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.396 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.396 DEBUG:boto:Params: {} 2023-09-29T22:22:54.396 DEBUG:boto:Token: None 2023-09-29T22:22:54.397 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO 2023-09-29T22:22:54.397 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI= 2023-09-29T22:22:54.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4EP79S8abnkP3Qx/EOByugXF6OI=', 'Content-Length': '0'} 2023-09-29T22:22:54.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '9Y3j-oRs0ga4D7AU-cCf3.-0jETmLhO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b30a96202bfe2ee-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.399 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.400 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.400 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.400 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.400 DEBUG:boto:Method: GET 2023-09-29T22:22:54.400 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.400 DEBUG:boto:Data: 2023-09-29T22:22:54.401 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.401 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.401 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.401 DEBUG:boto:Params: {} 2023-09-29T22:22:54.401 DEBUG:boto:Token: None 2023-09-29T22:22:54.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbd770522ad4bc1-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.446 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.447 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.447 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.447 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.447 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.447 DEBUG:boto:Method: GET 2023-09-29T22:22:54.447 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.447 DEBUG:boto:Data: 2023-09-29T22:22:54.447 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.448 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.448 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.448 DEBUG:boto:Params: {} 2023-09-29T22:22:54.448 DEBUG:boto:Token: None 2023-09-29T22:22:54.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f2ba2d8571ccf5-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.451 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.451 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.451 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.451 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.451 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.452 DEBUG:boto:Data: 2023-09-29T22:22:54.452 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.452 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.452 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.452 DEBUG:boto:Params: {} 2023-09-29T22:22:54.452 DEBUG:boto:Token: None 2023-09-29T22:22:54.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6ffc6b9c8a46f42-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.456 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.456 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.456 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.456 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.456 DEBUG:boto:Data: 2023-09-29T22:22:54.456 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.456 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.456 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.456 DEBUG:boto:Params: {} 2023-09-29T22:22:54.457 DEBUG:boto:Token: None 2023-09-29T22:22:54.457 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.457 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e039022ffcafa25-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.460 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.460 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.460 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.460 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.461 DEBUG:boto:Method: GET 2023-09-29T22:22:54.461 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.461 DEBUG:boto:Data: 2023-09-29T22:22:54.461 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.461 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.461 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.461 DEBUG:boto:Params: {} 2023-09-29T22:22:54.461 DEBUG:boto:Token: None 2023-09-29T22:22:54.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000958ee5bc5ae042f0-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.506 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.506 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.507 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.507 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.507 DEBUG:boto:Method: GET 2023-09-29T22:22:54.507 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.507 DEBUG:boto:Data: 2023-09-29T22:22:54.507 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.507 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.507 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.507 DEBUG:boto:Params: {} 2023-09-29T22:22:54.507 DEBUG:boto:Token: None 2023-09-29T22:22:54.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005784591e05dd7d09-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.550 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.550 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.551 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.551 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.551 DEBUG:boto:Method: GET 2023-09-29T22:22:54.551 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.551 DEBUG:boto:Data: 2023-09-29T22:22:54.551 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.551 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.551 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.551 DEBUG:boto:Params: {} 2023-09-29T22:22:54.551 DEBUG:boto:Token: None 2023-09-29T22:22:54.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6df1b84e7b12c2-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.598 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.599 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.599 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.599 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.599 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.599 DEBUG:boto:Method: GET 2023-09-29T22:22:54.599 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.599 DEBUG:boto:Data: 2023-09-29T22:22:54.600 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.600 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.600 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.600 DEBUG:boto:Params: {} 2023-09-29T22:22:54.600 DEBUG:boto:Token: None 2023-09-29T22:22:54.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.600 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006da006f713d42116-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.603 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.603 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.604 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.604 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.604 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.604 DEBUG:boto:Method: GET 2023-09-29T22:22:54.604 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.604 DEBUG:boto:Data: 2023-09-29T22:22:54.604 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.604 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.604 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.604 DEBUG:boto:Params: {} 2023-09-29T22:22:54.604 DEBUG:boto:Token: None 2023-09-29T22:22:54.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ= 2023-09-29T22:22:54.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ=', 'Content-Length': '0'} 2023-09-29T22:22:54.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b5db8713e7a7790-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.607 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.607 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.607 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.607 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.607 DEBUG:boto:Method: GET 2023-09-29T22:22:54.607 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.607 DEBUG:boto:Data: 2023-09-29T22:22:54.608 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.608 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.608 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.608 DEBUG:boto:Params: {} 2023-09-29T22:22:54.608 DEBUG:boto:Token: None 2023-09-29T22:22:54.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.608 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ= 2023-09-29T22:22:54.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ=', 'Content-Length': '0'} 2023-09-29T22:22:54.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000576796c7461b0c6b-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.650 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.650 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.651 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.651 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.651 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.651 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.651 DEBUG:boto:Data: 2023-09-29T22:22:54.651 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.651 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.651 DEBUG:boto:Params: {} 2023-09-29T22:22:54.651 DEBUG:boto:Token: None 2023-09-29T22:22:54.652 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LK/dnUSTl7fANfhQbASiMB+2PUs= 2023-09-29T22:22:54.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LK/dnUSTl7fANfhQbASiMB+2PUs=', 'Content-Length': '0'} 2023-09-29T22:22:54.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7f86daa4df0bcae-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.654 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.654 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.654 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.654 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.654 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.654 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.654 DEBUG:boto:Data: 2023-09-29T22:22:54.654 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.654 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.655 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.655 DEBUG:boto:Params: {} 2023-09-29T22:22:54.655 DEBUG:boto:Token: None 2023-09-29T22:22:54.655 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LK/dnUSTl7fANfhQbASiMB+2PUs= 2023-09-29T22:22:54.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LK/dnUSTl7fANfhQbASiMB+2PUs=', 'Content-Length': '0'} 2023-09-29T22:22:54.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e0618310c2515ba-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.657 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.657 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.657 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.657 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.657 DEBUG:boto:Method: GET 2023-09-29T22:22:54.658 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.658 DEBUG:boto:Data: 2023-09-29T22:22:54.658 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.658 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.658 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.658 DEBUG:boto:Params: {} 2023-09-29T22:22:54.658 DEBUG:boto:Token: None 2023-09-29T22:22:54.658 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.658 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ= 2023-09-29T22:22:54.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ=', 'Content-Length': '0'} 2023-09-29T22:22:54.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000890e55af73ffeb5b-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.706 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.706 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.707 DEBUG:boto:path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.707 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.707 DEBUG:boto:Method: GET 2023-09-29T22:22:54.707 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.707 DEBUG:boto:Data: 2023-09-29T22:22:54.707 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.707 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.707 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.707 DEBUG:boto:Params: {} 2023-09-29T22:22:54.707 DEBUG:boto:Token: None 2023-09-29T22:22:54.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF 2023-09-29T22:22:54.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ= 2023-09-29T22:22:54.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqSHQsFUxEAF6Xnvu0PpMZhb2yQ=', 'Content-Length': '0'} 2023-09-29T22:22:54.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'CwIobdwlPKtbIJWJbuSlt9ZJ6IK7upF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e0dd851a5b392b7-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.754 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.754 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.754 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.755 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.755 DEBUG:boto:Method: GET 2023-09-29T22:22:54.755 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.755 DEBUG:boto:Data: 2023-09-29T22:22:54.755 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.755 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.755 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.755 DEBUG:boto:Params: {} 2023-09-29T22:22:54.755 DEBUG:boto:Token: None 2023-09-29T22:22:54.755 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.756 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d367dbf54511b8-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.759 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.759 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.759 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.759 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.759 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.759 DEBUG:boto:Method: GET 2023-09-29T22:22:54.759 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.760 DEBUG:boto:Data: 2023-09-29T22:22:54.760 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.760 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.760 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.760 DEBUG:boto:Params: {} 2023-09-29T22:22:54.760 DEBUG:boto:Token: None 2023-09-29T22:22:54.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a04c0dd741174dd-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.806 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.807 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.807 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.807 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.807 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.807 DEBUG:boto:Data: 2023-09-29T22:22:54.807 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.807 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.808 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.808 DEBUG:boto:Params: {} 2023-09-29T22:22:54.808 DEBUG:boto:Token: None 2023-09-29T22:22:54.808 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.808 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fb628d3418d5fff-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.812 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.812 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.812 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.812 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:54.812 DEBUG:boto:Data: 2023-09-29T22:22:54.812 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.812 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.812 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.812 DEBUG:boto:Params: {} 2023-09-29T22:22:54.813 DEBUG:boto:Token: None 2023-09-29T22:22:54.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj 2023-09-29T22:22:54.813 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U= 2023-09-29T22:22:54.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fhws9CuM12lTiaYOvIkq7Tg835U=', 'Content-Length': '0'} 2023-09-29T22:22:54.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000397dc134783874f1-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.816 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.816 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.817 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.817 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.817 DEBUG:boto:Method: GET 2023-09-29T22:22:54.817 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.817 DEBUG:boto:Data: 2023-09-29T22:22:54.817 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.817 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.817 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.817 DEBUG:boto:Params: {} 2023-09-29T22:22:54.818 DEBUG:boto:Token: None 2023-09-29T22:22:54.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.818 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007811021268fbd43f-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.862 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.862 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.863 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.863 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.863 DEBUG:boto:Method: GET 2023-09-29T22:22:54.863 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.863 DEBUG:boto:Data: 2023-09-29T22:22:54.863 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.863 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.863 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.864 DEBUG:boto:Params: {} 2023-09-29T22:22:54.864 DEBUG:boto:Token: None 2023-09-29T22:22:54.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:54.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI= 2023-09-29T22:22:54.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e952ZarSQVv6kG6CJ5Ms/XlnAbI=', 'Content-Length': '0'} 2023-09-29T22:22:54.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e618c4d92ff8de5-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.906 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.907 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.907 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.907 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.907 DEBUG:boto:Method: GET 2023-09-29T22:22:54.907 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.907 DEBUG:boto:Data: 2023-09-29T22:22:54.907 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.907 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.907 DEBUG:boto:Params: {} 2023-09-29T22:22:54.908 DEBUG:boto:Token: None 2023-09-29T22:22:54.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c936776708f53c7-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.911 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.911 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:54.911 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:54.911 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.911 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.912 DEBUG:boto:Method: GET 2023-09-29T22:22:54.912 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:54.912 DEBUG:boto:Data: 2023-09-29T22:22:54.912 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.912 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.912 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.912 DEBUG:boto:Params: {} 2023-09-29T22:22:54.912 DEBUG:boto:Token: None 2023-09-29T22:22:54.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/ 2023-09-29T22:22:54.913 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss= 2023-09-29T22:22:54.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:luNL0vZAG/K0sDdBTB1YYRDxRss=', 'Content-Length': '0'} 2023-09-29T22:22:54.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8fa77b723ac4c79-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.962 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:54.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.963 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.963 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.963 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.963 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.963 DEBUG:boto:Method: GET 2023-09-29T22:22:54.963 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.964 DEBUG:boto:Data: 2023-09-29T22:22:54.964 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.964 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.964 DEBUG:boto:Params: {} 2023-09-29T22:22:54.964 DEBUG:boto:Token: None 2023-09-29T22:22:54.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM= 2023-09-29T22:22:54.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM=', 'Content-Length': '0'} 2023-09-29T22:22:54.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f097fb45d1873d28-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.967 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.967 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.967 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.967 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.967 DEBUG:boto:Method: GET 2023-09-29T22:22:54.968 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.968 DEBUG:boto:Data: 2023-09-29T22:22:54.968 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.968 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.968 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.968 DEBUG:boto:Params: {} 2023-09-29T22:22:54.969 DEBUG:boto:Token: None 2023-09-29T22:22:54.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM= 2023-09-29T22:22:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM=', 'Content-Length': '0'} 2023-09-29T22:22:54.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002da802252baa3fd2-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.971 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.972 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.972 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.972 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.972 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.972 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.972 DEBUG:boto:Data: 2023-09-29T22:22:54.972 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.972 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.972 DEBUG:boto:Params: {} 2023-09-29T22:22:54.973 DEBUG:boto:Token: None 2023-09-29T22:22:54.973 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P/9Al4cknlpCRIO/1vZYvoFgWRA= 2023-09-29T22:22:54.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P/9Al4cknlpCRIO/1vZYvoFgWRA=', 'Content-Length': '0'} 2023-09-29T22:22:54.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058798ff147cd91a8-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.975 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.975 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.975 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.975 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.975 DEBUG:boto:Method: HEAD 2023-09-29T22:22:54.975 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.975 DEBUG:boto:Data: 2023-09-29T22:22:54.976 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:54.976 DEBUG:boto:Port: 8000 2023-09-29T22:22:54.976 DEBUG:boto:Params: {} 2023-09-29T22:22:54.976 DEBUG:boto:Token: None 2023-09-29T22:22:54.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.976 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P/9Al4cknlpCRIO/1vZYvoFgWRA= 2023-09-29T22:22:54.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P/9Al4cknlpCRIO/1vZYvoFgWRA=', 'Content-Length': '0'} 2023-09-29T22:22:54.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060dc54c23f1171ac-0065174e3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:54.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:54.979 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:54.979 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:54.979 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.979 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.979 DEBUG:boto:Method: GET 2023-09-29T22:22:54.979 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.979 DEBUG:boto:Data: 2023-09-29T22:22:54.979 DEBUG:boto:Headers: {} 2023-09-29T22:22:54.979 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:54.979 DEBUG:boto:Port: 8001 2023-09-29T22:22:54.980 DEBUG:boto:Params: {} 2023-09-29T22:22:54.980 DEBUG:boto:Token: None 2023-09-29T22:22:54.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:54 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:54.980 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM= 2023-09-29T22:22:54.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:pK0sLAaoi2/C5hD8Cv+aww270JM=', 'Content-Length': '0'} 2023-09-29T22:22:54.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b75cdffd8783be0-0065174e3e-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.026 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.026 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.027 DEBUG:boto:path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:55.027 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:55.027 DEBUG:boto:Method: GET 2023-09-29T22:22:55.027 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:55.027 DEBUG:boto:Data: 2023-09-29T22:22:55.027 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.027 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.027 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.027 DEBUG:boto:Params: {} 2023-09-29T22:22:55.027 DEBUG:boto:Token: None 2023-09-29T22:22:55.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=hg.3uHbA9vEsXztaQyye1kYMtpgUPV4 2023-09-29T22:22:55.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6g6cTMmeQDlcPrcUxcG+R1gN1Eo= 2023-09-29T22:22:55.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6g6cTMmeQDlcPrcUxcG+R1gN1Eo=', 'Content-Length': '0'} 2023-09-29T22:22:55.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'hg.3uHbA9vEsXztaQyye1kYMtpgUPV4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b8238bb56ae1fa1-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.070 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.070 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.071 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.071 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.071 DEBUG:boto:Method: GET 2023-09-29T22:22:55.071 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.071 DEBUG:boto:Data: 2023-09-29T22:22:55.071 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.071 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.071 DEBUG:boto:Params: {} 2023-09-29T22:22:55.071 DEBUG:boto:Token: None 2023-09-29T22:22:55.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ca4924e5e81f34-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.074 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.075 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.075 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.075 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.075 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.075 DEBUG:boto:Method: GET 2023-09-29T22:22:55.075 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.075 DEBUG:boto:Data: 2023-09-29T22:22:55.075 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.076 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.076 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.076 DEBUG:boto:Params: {} 2023-09-29T22:22:55.076 DEBUG:boto:Token: None 2023-09-29T22:22:55.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a3c7a28e7fe9700-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.122 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.123 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.123 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.123 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.123 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.123 DEBUG:boto:Data: 2023-09-29T22:22:55.123 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.123 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.123 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.124 DEBUG:boto:Params: {} 2023-09-29T22:22:55.124 DEBUG:boto:Token: None 2023-09-29T22:22:55.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6a3396cb194dd92-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.127 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.127 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.127 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.128 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.128 DEBUG:boto:Data: 2023-09-29T22:22:55.128 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.128 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.128 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.128 DEBUG:boto:Params: {} 2023-09-29T22:22:55.128 DEBUG:boto:Token: None 2023-09-29T22:22:55.128 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f17f633401371f-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.132 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.132 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.132 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.132 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.132 DEBUG:boto:Method: GET 2023-09-29T22:22:55.132 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.132 DEBUG:boto:Data: 2023-09-29T22:22:55.132 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.132 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.133 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.133 DEBUG:boto:Params: {} 2023-09-29T22:22:55.133 DEBUG:boto:Token: None 2023-09-29T22:22:55.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.133 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000249053c06c941e4c-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.182 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.183 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.183 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.183 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.183 DEBUG:boto:Method: GET 2023-09-29T22:22:55.183 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.183 DEBUG:boto:Data: 2023-09-29T22:22:55.183 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.183 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.183 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.183 DEBUG:boto:Params: {} 2023-09-29T22:22:55.183 DEBUG:boto:Token: None 2023-09-29T22:22:55.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003638750b10373117-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.226 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.226 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.226 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.227 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.227 DEBUG:boto:Method: GET 2023-09-29T22:22:55.227 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.227 DEBUG:boto:Data: 2023-09-29T22:22:55.227 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.227 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.227 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.227 DEBUG:boto:Params: {} 2023-09-29T22:22:55.227 DEBUG:boto:Token: None 2023-09-29T22:22:55.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b1b1129cfd659d-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.274 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.275 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.275 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.275 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.275 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.275 DEBUG:boto:Method: GET 2023-09-29T22:22:55.275 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.275 DEBUG:boto:Data: 2023-09-29T22:22:55.275 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.275 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.276 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.276 DEBUG:boto:Params: {} 2023-09-29T22:22:55.276 DEBUG:boto:Token: None 2023-09-29T22:22:55.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a764413ab064693-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.322 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.323 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.323 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.323 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.323 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.323 DEBUG:boto:Method: GET 2023-09-29T22:22:55.323 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.323 DEBUG:boto:Data: 2023-09-29T22:22:55.323 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.324 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.324 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.324 DEBUG:boto:Params: {} 2023-09-29T22:22:55.324 DEBUG:boto:Token: None 2023-09-29T22:22:55.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.324 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag= 2023-09-29T22:22:55.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag=', 'Content-Length': '0'} 2023-09-29T22:22:55.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d79bfdb3260e2f6-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.326 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.326 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.327 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.327 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.327 DEBUG:boto:Method: GET 2023-09-29T22:22:55.327 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.327 DEBUG:boto:Data: 2023-09-29T22:22:55.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.327 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.327 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.327 DEBUG:boto:Params: {} 2023-09-29T22:22:55.327 DEBUG:boto:Token: None 2023-09-29T22:22:55.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag= 2023-09-29T22:22:55.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag=', 'Content-Length': '0'} 2023-09-29T22:22:55.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8fc8215ef924b6b-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.330 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.330 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.330 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.330 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.330 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.330 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.331 DEBUG:boto:Data: 2023-09-29T22:22:55.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.331 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.331 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.331 DEBUG:boto:Params: {} 2023-09-29T22:22:55.331 DEBUG:boto:Token: None 2023-09-29T22:22:55.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n9i2Vq+u447zb7nsGCNuJo0xiDM= 2023-09-29T22:22:55.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n9i2Vq+u447zb7nsGCNuJo0xiDM=', 'Content-Length': '0'} 2023-09-29T22:22:55.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000372578f39dba4cbf-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.333 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.333 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.334 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.334 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.334 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.334 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.334 DEBUG:boto:Data: 2023-09-29T22:22:55.334 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.334 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.334 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.334 DEBUG:boto:Params: {} 2023-09-29T22:22:55.335 DEBUG:boto:Token: None 2023-09-29T22:22:55.335 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.335 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n9i2Vq+u447zb7nsGCNuJo0xiDM= 2023-09-29T22:22:55.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n9i2Vq+u447zb7nsGCNuJo0xiDM=', 'Content-Length': '0'} 2023-09-29T22:22:55.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009425b3bc1de18bd8-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.337 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.337 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.337 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.337 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.337 DEBUG:boto:Method: GET 2023-09-29T22:22:55.337 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.338 DEBUG:boto:Data: 2023-09-29T22:22:55.338 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.338 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.338 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.338 DEBUG:boto:Params: {} 2023-09-29T22:22:55.338 DEBUG:boto:Token: None 2023-09-29T22:22:55.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag= 2023-09-29T22:22:55.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag=', 'Content-Length': '0'} 2023-09-29T22:22:55.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd7b76e073880df1-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.340 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.341 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.341 DEBUG:boto:path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.341 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.341 DEBUG:boto:Method: GET 2023-09-29T22:22:55.341 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.341 DEBUG:boto:Data: 2023-09-29T22:22:55.341 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.341 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.341 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.341 DEBUG:boto:Params: {} 2023-09-29T22:22:55.342 DEBUG:boto:Token: None 2023-09-29T22:22:55.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=FfTbXU5TRCNthRqwusMgCllOl6aNgs9 2023-09-29T22:22:55.342 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag= 2023-09-29T22:22:55.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VKS2ykb2Vgqecc7Ktfqcmrjrsag=', 'Content-Length': '0'} 2023-09-29T22:22:55.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'FfTbXU5TRCNthRqwusMgCllOl6aNgs9'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000540769df955355e4-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.386 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.386 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.387 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.387 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.387 DEBUG:boto:Method: GET 2023-09-29T22:22:55.387 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.387 DEBUG:boto:Data: 2023-09-29T22:22:55.387 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.387 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.387 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.387 DEBUG:boto:Params: {} 2023-09-29T22:22:55.388 DEBUG:boto:Token: None 2023-09-29T22:22:55.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078af7cc82f966f7e-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.391 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.391 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.392 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.392 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.392 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.392 DEBUG:boto:Method: GET 2023-09-29T22:22:55.392 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.392 DEBUG:boto:Data: 2023-09-29T22:22:55.392 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.392 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.392 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.392 DEBUG:boto:Params: {} 2023-09-29T22:22:55.393 DEBUG:boto:Token: None 2023-09-29T22:22:55.393 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.393 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1fb52ad6df72058-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.395 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.396 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.396 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.396 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.396 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.396 DEBUG:boto:Data: 2023-09-29T22:22:55.396 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.396 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.397 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.397 DEBUG:boto:Params: {} 2023-09-29T22:22:55.397 DEBUG:boto:Token: None 2023-09-29T22:22:55.397 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.397 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078b41197090f1eb3-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.400 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.400 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.401 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.401 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.401 DEBUG:boto:Data: 2023-09-29T22:22:55.401 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.401 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.401 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.401 DEBUG:boto:Params: {} 2023-09-29T22:22:55.401 DEBUG:boto:Token: None 2023-09-29T22:22:55.401 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.402 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eec9c981c8758ca6-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.405 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.405 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.405 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.405 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.405 DEBUG:boto:Method: GET 2023-09-29T22:22:55.405 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.406 DEBUG:boto:Data: 2023-09-29T22:22:55.406 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.406 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.406 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.406 DEBUG:boto:Params: {} 2023-09-29T22:22:55.406 DEBUG:boto:Token: None 2023-09-29T22:22:55.406 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.406 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048939aac02d20ac2-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.450 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.450 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.451 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.451 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.451 DEBUG:boto:Method: GET 2023-09-29T22:22:55.451 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.451 DEBUG:boto:Data: 2023-09-29T22:22:55.451 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.451 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.451 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.451 DEBUG:boto:Params: {} 2023-09-29T22:22:55.452 DEBUG:boto:Token: None 2023-09-29T22:22:55.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.452 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d3aab8998beaa87-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.498 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.499 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.499 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.499 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.499 DEBUG:boto:Method: GET 2023-09-29T22:22:55.499 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.499 DEBUG:boto:Data: 2023-09-29T22:22:55.499 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.499 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.499 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.500 DEBUG:boto:Params: {} 2023-09-29T22:22:55.500 DEBUG:boto:Token: None 2023-09-29T22:22:55.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaca6faff5af1bd8-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.503 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.503 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.504 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.504 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.504 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.504 DEBUG:boto:Method: GET 2023-09-29T22:22:55.504 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.504 DEBUG:boto:Data: 2023-09-29T22:22:55.504 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.504 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.504 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.504 DEBUG:boto:Params: {} 2023-09-29T22:22:55.505 DEBUG:boto:Token: None 2023-09-29T22:22:55.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791bcac23acbe831-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.507 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.508 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.508 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.508 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.508 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.508 DEBUG:boto:Method: GET 2023-09-29T22:22:55.509 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.509 DEBUG:boto:Data: 2023-09-29T22:22:55.509 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.509 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.509 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.509 DEBUG:boto:Params: {} 2023-09-29T22:22:55.509 DEBUG:boto:Token: None 2023-09-29T22:22:55.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ= 2023-09-29T22:22:55.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ=', 'Content-Length': '0'} 2023-09-29T22:22:55.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e58e3a4a231589a-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.512 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.512 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.512 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.512 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.512 DEBUG:boto:Method: GET 2023-09-29T22:22:55.512 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.512 DEBUG:boto:Data: 2023-09-29T22:22:55.512 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.512 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.512 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.513 DEBUG:boto:Params: {} 2023-09-29T22:22:55.513 DEBUG:boto:Token: None 2023-09-29T22:22:55.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ= 2023-09-29T22:22:55.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ=', 'Content-Length': '0'} 2023-09-29T22:22:55.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e856c3798f1f6e13-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.515 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.515 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.515 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.515 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.515 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.517 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.517 DEBUG:boto:Data: 2023-09-29T22:22:55.518 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.518 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.518 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.518 DEBUG:boto:Params: {} 2023-09-29T22:22:55.518 DEBUG:boto:Token: None 2023-09-29T22:22:55.518 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.518 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PnOHv3QKyn1YUhtGf01/tJIGZ68= 2023-09-29T22:22:55.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PnOHv3QKyn1YUhtGf01/tJIGZ68=', 'Content-Length': '0'} 2023-09-29T22:22:55.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000547b2343eea55ae8-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.520 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.521 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.521 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.521 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.521 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.521 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.521 DEBUG:boto:Data: 2023-09-29T22:22:55.521 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.521 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.521 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.522 DEBUG:boto:Params: {} 2023-09-29T22:22:55.522 DEBUG:boto:Token: None 2023-09-29T22:22:55.522 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.522 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PnOHv3QKyn1YUhtGf01/tJIGZ68= 2023-09-29T22:22:55.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PnOHv3QKyn1YUhtGf01/tJIGZ68=', 'Content-Length': '0'} 2023-09-29T22:22:55.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000884a658df77b5c63-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.524 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.524 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.524 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.524 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.525 DEBUG:boto:Method: GET 2023-09-29T22:22:55.525 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.525 DEBUG:boto:Data: 2023-09-29T22:22:55.525 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.525 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.525 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.525 DEBUG:boto:Params: {} 2023-09-29T22:22:55.525 DEBUG:boto:Token: None 2023-09-29T22:22:55.525 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.525 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ= 2023-09-29T22:22:55.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ=', 'Content-Length': '0'} 2023-09-29T22:22:55.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f6129686a47259a-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.570 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.570 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.571 DEBUG:boto:path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.571 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.571 DEBUG:boto:Method: GET 2023-09-29T22:22:55.571 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.571 DEBUG:boto:Data: 2023-09-29T22:22:55.571 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.571 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.571 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.571 DEBUG:boto:Params: {} 2023-09-29T22:22:55.572 DEBUG:boto:Token: None 2023-09-29T22:22:55.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC 2023-09-29T22:22:55.572 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ= 2023-09-29T22:22:55.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Tqzp4Kch6eosvA00uIWhHGD+mqQ=', 'Content-Length': '0'} 2023-09-29T22:22:55.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'QjR0iuHZYzmF5y5FQx1VE5-Jsaa5WtC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000567d4a899d5d478e-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.574 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.574 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.574 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.574 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.574 DEBUG:boto:Method: GET 2023-09-29T22:22:55.575 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.575 DEBUG:boto:Data: 2023-09-29T22:22:55.575 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.575 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.575 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.575 DEBUG:boto:Params: {} 2023-09-29T22:22:55.575 DEBUG:boto:Token: None 2023-09-29T22:22:55.575 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.575 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab20d0fbc26fd7eb-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.618 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.619 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.619 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.619 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.619 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.619 DEBUG:boto:Method: GET 2023-09-29T22:22:55.619 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.619 DEBUG:boto:Data: 2023-09-29T22:22:55.619 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.620 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.620 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.620 DEBUG:boto:Params: {} 2023-09-29T22:22:55.620 DEBUG:boto:Token: None 2023-09-29T22:22:55.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019be923175e5f2dd-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.623 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.623 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.623 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.623 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.623 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.624 DEBUG:boto:Data: 2023-09-29T22:22:55.624 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.624 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.624 DEBUG:boto:Params: {} 2023-09-29T22:22:55.624 DEBUG:boto:Token: None 2023-09-29T22:22:55.624 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000959c67ffc8707501-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.628 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.628 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.628 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.628 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:55.628 DEBUG:boto:Data: 2023-09-29T22:22:55.628 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.628 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.629 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.629 DEBUG:boto:Params: {} 2023-09-29T22:22:55.629 DEBUG:boto:Token: None 2023-09-29T22:22:55.629 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj 2023-09-29T22:22:55.629 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY= 2023-09-29T22:22:55.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kyySbm7OBiM1t20miYQtnqaVmmY=', 'Content-Length': '0'} 2023-09-29T22:22:55.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aecb4bf48eb4fa8-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.633 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.633 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.633 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.633 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.633 DEBUG:boto:Method: GET 2023-09-29T22:22:55.633 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.633 DEBUG:boto:Data: 2023-09-29T22:22:55.633 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.633 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.633 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.633 DEBUG:boto:Params: {} 2023-09-29T22:22:55.634 DEBUG:boto:Token: None 2023-09-29T22:22:55.634 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.634 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6e57fa1cea9939f-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.678 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.678 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.679 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.679 DEBUG:boto:Method: GET 2023-09-29T22:22:55.679 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.679 DEBUG:boto:Data: 2023-09-29T22:22:55.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.679 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.679 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.679 DEBUG:boto:Params: {} 2023-09-29T22:22:55.680 DEBUG:boto:Token: None 2023-09-29T22:22:55.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:55.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw= 2023-09-29T22:22:55.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lUhhyAEdpL105bXMuuK3doqFJbw=', 'Content-Length': '0'} 2023-09-29T22:22:55.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006125e68ff9f1bfde-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.722 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.722 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.723 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.723 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.723 DEBUG:boto:Method: GET 2023-09-29T22:22:55.723 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.723 DEBUG:boto:Data: 2023-09-29T22:22:55.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.723 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.723 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.723 DEBUG:boto:Params: {} 2023-09-29T22:22:55.723 DEBUG:boto:Token: None 2023-09-29T22:22:55.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff924bd5bd55f90-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.770 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.771 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.771 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.771 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.771 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.771 DEBUG:boto:Method: GET 2023-09-29T22:22:55.771 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.771 DEBUG:boto:Data: 2023-09-29T22:22:55.771 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.772 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.772 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.772 DEBUG:boto:Params: {} 2023-09-29T22:22:55.772 DEBUG:boto:Token: None 2023-09-29T22:22:55.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.773 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4c4bba24cfb6a1-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.818 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:55.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.819 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.819 DEBUG:boto:Method: GET 2023-09-29T22:22:55.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.819 DEBUG:boto:Data: 2023-09-29T22:22:55.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.820 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.820 DEBUG:boto:Params: {} 2023-09-29T22:22:55.820 DEBUG:boto:Token: None 2023-09-29T22:22:55.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA= 2023-09-29T22:22:55.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA=', 'Content-Length': '0'} 2023-09-29T22:22:55.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001429884a94bfdb5d-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.823 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.823 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.823 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.823 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.823 DEBUG:boto:Method: GET 2023-09-29T22:22:55.823 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.823 DEBUG:boto:Data: 2023-09-29T22:22:55.823 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.823 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.824 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.824 DEBUG:boto:Params: {} 2023-09-29T22:22:55.824 DEBUG:boto:Token: None 2023-09-29T22:22:55.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA= 2023-09-29T22:22:55.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA=', 'Content-Length': '0'} 2023-09-29T22:22:55.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000560bd030eaf4a005-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.866 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.866 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.867 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.867 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.867 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.867 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.867 DEBUG:boto:Data: 2023-09-29T22:22:55.867 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.867 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.867 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.867 DEBUG:boto:Params: {} 2023-09-29T22:22:55.867 DEBUG:boto:Token: None 2023-09-29T22:22:55.868 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.868 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vPNCg3EthRiz9Hth8JMEQxuw/1c= 2023-09-29T22:22:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vPNCg3EthRiz9Hth8JMEQxuw/1c=', 'Content-Length': '0'} 2023-09-29T22:22:55.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aeb43cf2bb525be-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.870 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.870 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.870 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.870 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.870 DEBUG:boto:Method: HEAD 2023-09-29T22:22:55.870 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.870 DEBUG:boto:Data: 2023-09-29T22:22:55.871 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.871 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.871 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.871 DEBUG:boto:Params: {} 2023-09-29T22:22:55.871 DEBUG:boto:Token: None 2023-09-29T22:22:55.871 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.871 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vPNCg3EthRiz9Hth8JMEQxuw/1c= 2023-09-29T22:22:55.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vPNCg3EthRiz9Hth8JMEQxuw/1c=', 'Content-Length': '0'} 2023-09-29T22:22:55.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045d798fd039cfe5e-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:55.873 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.874 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.874 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.874 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.874 DEBUG:boto:Method: GET 2023-09-29T22:22:55.874 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.874 DEBUG:boto:Data: 2023-09-29T22:22:55.874 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.874 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.874 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.874 DEBUG:boto:Params: {} 2023-09-29T22:22:55.875 DEBUG:boto:Token: None 2023-09-29T22:22:55.875 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.875 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA= 2023-09-29T22:22:55.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA=', 'Content-Length': '0'} 2023-09-29T22:22:55.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e216a54d7125e94c-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.922 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:55.923 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:55.923 DEBUG:boto:path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.923 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.923 DEBUG:boto:Method: GET 2023-09-29T22:22:55.923 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.923 DEBUG:boto:Data: 2023-09-29T22:22:55.923 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.923 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:55.923 DEBUG:boto:Port: 8000 2023-09-29T22:22:55.923 DEBUG:boto:Params: {} 2023-09-29T22:22:55.924 DEBUG:boto:Token: None 2023-09-29T22:22:55.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/obj?versionId=jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g 2023-09-29T22:22:55.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA= 2023-09-29T22:22:55.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q6lq36emm+9q9vqUgcPT7Y1X3gA=', 'Content-Length': '0'} 2023-09-29T22:22:55.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'jpJVcdJHvZoYOCaa43L2AAr7v0bG4-g'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc2cdfee9b937a6b-0065174e3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:55.966 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:55.967 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:55.967 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.967 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.967 DEBUG:boto:Method: GET 2023-09-29T22:22:55.967 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:55.967 DEBUG:boto:Data: 2023-09-29T22:22:55.967 DEBUG:boto:Headers: {} 2023-09-29T22:22:55.967 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:55.967 DEBUG:boto:Port: 8001 2023-09-29T22:22:55.967 DEBUG:boto:Params: {} 2023-09-29T22:22:55.968 DEBUG:boto:Token: None 2023-09-29T22:22:55.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:55 GMT /dmuhsu-19/ 2023-09-29T22:22:55.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw= 2023-09-29T22:22:55.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LpDmf944yeiH7HtAmsaUwXnEWkw=', 'Content-Length': '0'} 2023-09-29T22:22:55.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da47dea9dfcbb85-0065174e3f-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.014 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.015 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.015 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.015 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.015 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.016 DEBUG:boto:Method: GET 2023-09-29T22:22:56.016 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.016 DEBUG:boto:Data: 2023-09-29T22:22:56.016 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.016 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.016 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.016 DEBUG:boto:Params: {} 2023-09-29T22:22:56.016 DEBUG:boto:Token: None 2023-09-29T22:22:56.017 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.017 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008818b009f93a5808-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.062 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.063 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.063 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.063 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.063 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.063 DEBUG:boto:Data: 2023-09-29T22:22:56.064 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.064 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.064 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.064 DEBUG:boto:Params: {} 2023-09-29T22:22:56.064 DEBUG:boto:Token: None 2023-09-29T22:22:56.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b59c1c0d61a2fae-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.068 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.068 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.068 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.068 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.068 DEBUG:boto:Data: 2023-09-29T22:22:56.068 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.068 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.068 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.068 DEBUG:boto:Params: {} 2023-09-29T22:22:56.069 DEBUG:boto:Token: None 2023-09-29T22:22:56.069 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad94d729d7236f61-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.072 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.073 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.073 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.073 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.073 DEBUG:boto:Method: GET 2023-09-29T22:22:56.073 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.073 DEBUG:boto:Data: 2023-09-29T22:22:56.073 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.073 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.073 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.073 DEBUG:boto:Params: {} 2023-09-29T22:22:56.073 DEBUG:boto:Token: None 2023-09-29T22:22:56.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.074 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061eae02b7e9cbbc1-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.118 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.119 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.119 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.119 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.119 DEBUG:boto:Method: GET 2023-09-29T22:22:56.119 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.119 DEBUG:boto:Data: 2023-09-29T22:22:56.119 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.119 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.120 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.120 DEBUG:boto:Params: {} 2023-09-29T22:22:56.120 DEBUG:boto:Token: None 2023-09-29T22:22:56.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f154c0f4215010c-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.162 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.163 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.163 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.163 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.163 DEBUG:boto:Method: GET 2023-09-29T22:22:56.163 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.163 DEBUG:boto:Data: 2023-09-29T22:22:56.163 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.163 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.163 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.163 DEBUG:boto:Params: {} 2023-09-29T22:22:56.164 DEBUG:boto:Token: None 2023-09-29T22:22:56.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.164 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de312f48e56d7a80-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.210 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.211 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.211 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.211 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.211 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.211 DEBUG:boto:Method: GET 2023-09-29T22:22:56.211 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.211 DEBUG:boto:Data: 2023-09-29T22:22:56.212 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.212 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.212 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.212 DEBUG:boto:Params: {} 2023-09-29T22:22:56.212 DEBUG:boto:Token: None 2023-09-29T22:22:56.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a61c5df6084ac2af-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.258 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.259 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.259 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.259 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.259 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.259 DEBUG:boto:Method: GET 2023-09-29T22:22:56.260 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.260 DEBUG:boto:Data: 2023-09-29T22:22:56.260 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.260 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.260 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.260 DEBUG:boto:Params: {} 2023-09-29T22:22:56.260 DEBUG:boto:Token: None 2023-09-29T22:22:56.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM= 2023-09-29T22:22:56.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM=', 'Content-Length': '0'} 2023-09-29T22:22:56.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050703a1bae2e378d-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.263 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.263 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.263 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.263 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.263 DEBUG:boto:Method: GET 2023-09-29T22:22:56.263 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.263 DEBUG:boto:Data: 2023-09-29T22:22:56.263 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.263 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.264 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.264 DEBUG:boto:Params: {} 2023-09-29T22:22:56.264 DEBUG:boto:Token: None 2023-09-29T22:22:56.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM= 2023-09-29T22:22:56.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM=', 'Content-Length': '0'} 2023-09-29T22:22:56.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dda65b2d32c2177f-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.266 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.266 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.266 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.266 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.267 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.267 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.267 DEBUG:boto:Data: 2023-09-29T22:22:56.267 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.267 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.267 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.267 DEBUG:boto:Params: {} 2023-09-29T22:22:56.267 DEBUG:boto:Token: None 2023-09-29T22:22:56.267 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ycZUAL/1m5IcsewAauwTEtlPKtw= 2023-09-29T22:22:56.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ycZUAL/1m5IcsewAauwTEtlPKtw=', 'Content-Length': '0'} 2023-09-29T22:22:56.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ddf4514c2e3c54d-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.270 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.270 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.270 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.270 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.270 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.270 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.270 DEBUG:boto:Data: 2023-09-29T22:22:56.270 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.271 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.271 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.271 DEBUG:boto:Params: {} 2023-09-29T22:22:56.271 DEBUG:boto:Token: None 2023-09-29T22:22:56.271 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ycZUAL/1m5IcsewAauwTEtlPKtw= 2023-09-29T22:22:56.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ycZUAL/1m5IcsewAauwTEtlPKtw=', 'Content-Length': '0'} 2023-09-29T22:22:56.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a32990996f0c7d3-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.273 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.273 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.274 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.274 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.274 DEBUG:boto:Method: GET 2023-09-29T22:22:56.274 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.274 DEBUG:boto:Data: 2023-09-29T22:22:56.274 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.274 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.274 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.274 DEBUG:boto:Params: {} 2023-09-29T22:22:56.274 DEBUG:boto:Token: None 2023-09-29T22:22:56.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM= 2023-09-29T22:22:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM=', 'Content-Length': '0'} 2023-09-29T22:22:56.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000275d13cd020a4bee-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.277 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.277 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.277 DEBUG:boto:path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.277 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.277 DEBUG:boto:Method: GET 2023-09-29T22:22:56.277 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.277 DEBUG:boto:Data: 2023-09-29T22:22:56.277 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.278 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.278 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.278 DEBUG:boto:Params: {} 2023-09-29T22:22:56.278 DEBUG:boto:Token: None 2023-09-29T22:22:56.278 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka 2023-09-29T22:22:56.278 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM= 2023-09-29T22:22:56.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6+kdrHVdW5HKRiohL9Ji+3xu9qM=', 'Content-Length': '0'} 2023-09-29T22:22:56.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'HWlT9cq3Z9KfNRxOyf6NTGYD6.iFXka'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a2c6026a609d7e6-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.326 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.326 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.327 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.327 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.327 DEBUG:boto:Method: GET 2023-09-29T22:22:56.327 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.327 DEBUG:boto:Data: 2023-09-29T22:22:56.327 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.327 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.327 DEBUG:boto:Params: {} 2023-09-29T22:22:56.327 DEBUG:boto:Token: None 2023-09-29T22:22:56.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888fd0b1115f3dc8-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.330 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.331 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.331 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.331 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.331 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.331 DEBUG:boto:Method: GET 2023-09-29T22:22:56.331 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.331 DEBUG:boto:Data: 2023-09-29T22:22:56.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.332 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.332 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.332 DEBUG:boto:Params: {} 2023-09-29T22:22:56.332 DEBUG:boto:Token: None 2023-09-29T22:22:56.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.332 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f6d69fcb2e3f5a-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.335 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.335 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.335 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.335 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.335 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.335 DEBUG:boto:Data: 2023-09-29T22:22:56.335 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.336 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.336 DEBUG:boto:Params: {} 2023-09-29T22:22:56.336 DEBUG:boto:Token: None 2023-09-29T22:22:56.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.336 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e90fe6d281c0591-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.339 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.339 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.339 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.340 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.340 DEBUG:boto:Data: 2023-09-29T22:22:56.340 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.340 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.340 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.340 DEBUG:boto:Params: {} 2023-09-29T22:22:56.340 DEBUG:boto:Token: None 2023-09-29T22:22:56.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.340 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad543d6135c52004-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.346 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.346 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.346 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.346 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.346 DEBUG:boto:Method: GET 2023-09-29T22:22:56.346 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.346 DEBUG:boto:Data: 2023-09-29T22:22:56.346 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.347 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.347 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.347 DEBUG:boto:Params: {} 2023-09-29T22:22:56.347 DEBUG:boto:Token: None 2023-09-29T22:22:56.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015d06cb8f5e27080-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.390 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.390 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.391 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.391 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.391 DEBUG:boto:Method: GET 2023-09-29T22:22:56.391 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.391 DEBUG:boto:Data: 2023-09-29T22:22:56.391 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.391 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.391 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.391 DEBUG:boto:Params: {} 2023-09-29T22:22:56.391 DEBUG:boto:Token: None 2023-09-29T22:22:56.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.392 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039c60b2c0af77d46-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.434 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.435 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.435 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.435 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.435 DEBUG:boto:Method: GET 2023-09-29T22:22:56.435 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.435 DEBUG:boto:Data: 2023-09-29T22:22:56.435 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.435 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.435 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.435 DEBUG:boto:Params: {} 2023-09-29T22:22:56.435 DEBUG:boto:Token: None 2023-09-29T22:22:56.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.436 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e4e4a08128e002-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.438 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.439 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.439 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.439 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.439 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.439 DEBUG:boto:Method: GET 2023-09-29T22:22:56.439 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.439 DEBUG:boto:Data: 2023-09-29T22:22:56.440 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.440 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.440 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.440 DEBUG:boto:Params: {} 2023-09-29T22:22:56.440 DEBUG:boto:Token: None 2023-09-29T22:22:56.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d440542045c13cf-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.443 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.443 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.443 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.443 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.443 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.443 DEBUG:boto:Method: GET 2023-09-29T22:22:56.444 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.444 DEBUG:boto:Data: 2023-09-29T22:22:56.444 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.444 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.444 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.444 DEBUG:boto:Params: {} 2023-09-29T22:22:56.444 DEBUG:boto:Token: None 2023-09-29T22:22:56.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.444 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA= 2023-09-29T22:22:56.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA=', 'Content-Length': '0'} 2023-09-29T22:22:56.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5e19008bfeba97a-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.446 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.447 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.447 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.447 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.447 DEBUG:boto:Method: GET 2023-09-29T22:22:56.447 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.447 DEBUG:boto:Data: 2023-09-29T22:22:56.447 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.447 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.447 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.447 DEBUG:boto:Params: {} 2023-09-29T22:22:56.447 DEBUG:boto:Token: None 2023-09-29T22:22:56.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA= 2023-09-29T22:22:56.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA=', 'Content-Length': '0'} 2023-09-29T22:22:56.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa681ae0a7dcbb07-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.490 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.490 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.491 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.491 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.491 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.491 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.491 DEBUG:boto:Data: 2023-09-29T22:22:56.491 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.491 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.491 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.491 DEBUG:boto:Params: {} 2023-09-29T22:22:56.491 DEBUG:boto:Token: None 2023-09-29T22:22:56.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/EH7f0KWIJ9kxSTPUdALqX5GaqQ= 2023-09-29T22:22:56.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/EH7f0KWIJ9kxSTPUdALqX5GaqQ=', 'Content-Length': '0'} 2023-09-29T22:22:56.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee4fd6c3bdfde597-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.493 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.494 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.494 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.494 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.494 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.494 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.494 DEBUG:boto:Data: 2023-09-29T22:22:56.494 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.494 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.494 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.494 DEBUG:boto:Params: {} 2023-09-29T22:22:56.495 DEBUG:boto:Token: None 2023-09-29T22:22:56.495 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/EH7f0KWIJ9kxSTPUdALqX5GaqQ= 2023-09-29T22:22:56.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/EH7f0KWIJ9kxSTPUdALqX5GaqQ=', 'Content-Length': '0'} 2023-09-29T22:22:56.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9263f1957730f1-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.497 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.497 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.497 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.497 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.497 DEBUG:boto:Method: GET 2023-09-29T22:22:56.497 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.497 DEBUG:boto:Data: 2023-09-29T22:22:56.497 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.498 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.498 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.498 DEBUG:boto:Params: {} 2023-09-29T22:22:56.498 DEBUG:boto:Token: None 2023-09-29T22:22:56.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.498 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA= 2023-09-29T22:22:56.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA=', 'Content-Length': '0'} 2023-09-29T22:22:56.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019ac3e678fb95c66-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.542 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.543 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.544 DEBUG:boto:path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.545 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.545 DEBUG:boto:Method: GET 2023-09-29T22:22:56.545 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.545 DEBUG:boto:Data: 2023-09-29T22:22:56.545 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.545 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.545 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.545 DEBUG:boto:Params: {} 2023-09-29T22:22:56.546 DEBUG:boto:Token: None 2023-09-29T22:22:56.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=JJO1acwgSpskwqQWN.dkCFtTPz5ryQj 2023-09-29T22:22:56.546 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA= 2023-09-29T22:22:56.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:etlnxKn83OOBh4lV+hGfd4iXERA=', 'Content-Length': '0'} 2023-09-29T22:22:56.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'JJO1acwgSpskwqQWN.dkCFtTPz5ryQj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab7d250d1c2519d9-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.590 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.591 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.591 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.591 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.591 DEBUG:boto:Method: GET 2023-09-29T22:22:56.591 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.591 DEBUG:boto:Data: 2023-09-29T22:22:56.591 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.591 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.591 DEBUG:boto:Params: {} 2023-09-29T22:22:56.592 DEBUG:boto:Token: None 2023-09-29T22:22:56.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cbc4ec34abe4524-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.595 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.595 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.595 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.596 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.596 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.596 DEBUG:boto:Method: GET 2023-09-29T22:22:56.596 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.596 DEBUG:boto:Data: 2023-09-29T22:22:56.596 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.596 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.596 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.596 DEBUG:boto:Params: {} 2023-09-29T22:22:56.596 DEBUG:boto:Token: None 2023-09-29T22:22:56.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.597 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e106b87cfc5d0416-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.599 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.600 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.600 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.600 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.600 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.600 DEBUG:boto:Data: 2023-09-29T22:22:56.600 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.600 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.600 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.600 DEBUG:boto:Params: {} 2023-09-29T22:22:56.601 DEBUG:boto:Token: None 2023-09-29T22:22:56.601 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.601 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db1cbd640b3c5c25-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.604 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.604 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.604 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.604 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.605 DEBUG:boto:Data: 2023-09-29T22:22:56.605 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.605 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.605 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.605 DEBUG:boto:Params: {} 2023-09-29T22:22:56.605 DEBUG:boto:Token: None 2023-09-29T22:22:56.605 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe19b5011b11edb6-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.609 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.609 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.609 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.609 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.609 DEBUG:boto:Method: GET 2023-09-29T22:22:56.609 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.609 DEBUG:boto:Data: 2023-09-29T22:22:56.609 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.609 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.610 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.610 DEBUG:boto:Params: {} 2023-09-29T22:22:56.610 DEBUG:boto:Token: None 2023-09-29T22:22:56.610 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e1a6c3e79090f28-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.654 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.654 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.655 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.655 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.655 DEBUG:boto:Method: GET 2023-09-29T22:22:56.655 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.655 DEBUG:boto:Data: 2023-09-29T22:22:56.655 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.655 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.655 DEBUG:boto:Params: {} 2023-09-29T22:22:56.656 DEBUG:boto:Token: None 2023-09-29T22:22:56.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d0cdd0e7ad8be53-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.698 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.699 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.699 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.699 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.699 DEBUG:boto:Method: GET 2023-09-29T22:22:56.699 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.699 DEBUG:boto:Data: 2023-09-29T22:22:56.699 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.699 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.699 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.699 DEBUG:boto:Params: {} 2023-09-29T22:22:56.700 DEBUG:boto:Token: None 2023-09-29T22:22:56.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e351c3400c8febce-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.702 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.703 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.703 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.703 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.703 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.703 DEBUG:boto:Method: GET 2023-09-29T22:22:56.703 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.703 DEBUG:boto:Data: 2023-09-29T22:22:56.704 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.704 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.704 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.704 DEBUG:boto:Params: {} 2023-09-29T22:22:56.704 DEBUG:boto:Token: None 2023-09-29T22:22:56.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cdc6aae82507df8-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.707 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.707 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.707 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.708 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.708 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.708 DEBUG:boto:Method: GET 2023-09-29T22:22:56.708 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.708 DEBUG:boto:Data: 2023-09-29T22:22:56.708 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.708 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.708 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.708 DEBUG:boto:Params: {} 2023-09-29T22:22:56.708 DEBUG:boto:Token: None 2023-09-29T22:22:56.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4= 2023-09-29T22:22:56.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4=', 'Content-Length': '0'} 2023-09-29T22:22:56.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c6f6c787cb7c59e-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.711 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.711 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.711 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.711 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.711 DEBUG:boto:Method: GET 2023-09-29T22:22:56.711 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.712 DEBUG:boto:Data: 2023-09-29T22:22:56.712 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.712 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.712 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.712 DEBUG:boto:Params: {} 2023-09-29T22:22:56.712 DEBUG:boto:Token: None 2023-09-29T22:22:56.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4= 2023-09-29T22:22:56.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4=', 'Content-Length': '0'} 2023-09-29T22:22:56.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ac8d5b323690dec-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.714 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.714 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.715 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.715 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.715 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.715 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.715 DEBUG:boto:Data: 2023-09-29T22:22:56.715 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.715 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.715 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.715 DEBUG:boto:Params: {} 2023-09-29T22:22:56.715 DEBUG:boto:Token: None 2023-09-29T22:22:56.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mKKV3UcfHzWM5PJOtusC4f8SOeI= 2023-09-29T22:22:56.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mKKV3UcfHzWM5PJOtusC4f8SOeI=', 'Content-Length': '0'} 2023-09-29T22:22:56.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004edc959f2c54f472-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.718 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.718 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.718 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.718 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.718 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.718 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.719 DEBUG:boto:Data: 2023-09-29T22:22:56.719 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.719 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.719 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.719 DEBUG:boto:Params: {} 2023-09-29T22:22:56.719 DEBUG:boto:Token: None 2023-09-29T22:22:56.719 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.719 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mKKV3UcfHzWM5PJOtusC4f8SOeI= 2023-09-29T22:22:56.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mKKV3UcfHzWM5PJOtusC4f8SOeI=', 'Content-Length': '0'} 2023-09-29T22:22:56.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c905cc31132d77d9-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.721 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.721 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.722 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.722 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.722 DEBUG:boto:Method: GET 2023-09-29T22:22:56.722 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.722 DEBUG:boto:Data: 2023-09-29T22:22:56.722 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.722 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.722 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.722 DEBUG:boto:Params: {} 2023-09-29T22:22:56.722 DEBUG:boto:Token: None 2023-09-29T22:22:56.723 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.723 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4= 2023-09-29T22:22:56.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4=', 'Content-Length': '0'} 2023-09-29T22:22:56.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa27230a3640fa9c-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.766 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.766 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.767 DEBUG:boto:path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.767 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.767 DEBUG:boto:Method: GET 2023-09-29T22:22:56.767 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.767 DEBUG:boto:Data: 2023-09-29T22:22:56.767 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.767 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.767 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.767 DEBUG:boto:Params: {} 2023-09-29T22:22:56.767 DEBUG:boto:Token: None 2023-09-29T22:22:56.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=y84w72ok3GVudkBmY3Rlj4Nja225EFV 2023-09-29T22:22:56.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4= 2023-09-29T22:22:56.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSapKFojruHI2dzVGGhhsVt0jW4=', 'Content-Length': '0'} 2023-09-29T22:22:56.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'y84w72ok3GVudkBmY3Rlj4Nja225EFV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2edd46f4c76e965-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.770 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.770 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.770 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.770 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.770 DEBUG:boto:Method: GET 2023-09-29T22:22:56.770 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.770 DEBUG:boto:Data: 2023-09-29T22:22:56.771 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.771 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.771 DEBUG:boto:Params: {} 2023-09-29T22:22:56.771 DEBUG:boto:Token: None 2023-09-29T22:22:56.771 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943be2d7fe667e07-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.814 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.815 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.815 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.815 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.815 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.815 DEBUG:boto:Method: GET 2023-09-29T22:22:56.815 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.815 DEBUG:boto:Data: 2023-09-29T22:22:56.815 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.815 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.816 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.816 DEBUG:boto:Params: {} 2023-09-29T22:22:56.816 DEBUG:boto:Token: None 2023-09-29T22:22:56.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca6a68d8f693b46-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.819 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.819 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.819 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.819 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.819 DEBUG:boto:Data: 2023-09-29T22:22:56.820 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.820 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.820 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.820 DEBUG:boto:Params: {} 2023-09-29T22:22:56.820 DEBUG:boto:Token: None 2023-09-29T22:22:56.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ecbb3d8c3ea7960-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.823 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.824 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.824 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.824 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:56.824 DEBUG:boto:Data: 2023-09-29T22:22:56.824 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.824 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.824 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.824 DEBUG:boto:Params: {} 2023-09-29T22:22:56.824 DEBUG:boto:Token: None 2023-09-29T22:22:56.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj 2023-09-29T22:22:56.825 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U= 2023-09-29T22:22:56.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lwd9NZxxpSb5043HKChwZay0o3U=', 'Content-Length': '0'} 2023-09-29T22:22:56.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f43be1293d9c0bf-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.828 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.828 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.828 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.828 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.828 DEBUG:boto:Method: GET 2023-09-29T22:22:56.829 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.829 DEBUG:boto:Data: 2023-09-29T22:22:56.829 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.829 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.829 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.829 DEBUG:boto:Params: {} 2023-09-29T22:22:56.829 DEBUG:boto:Token: None 2023-09-29T22:22:56.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.829 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076138bb92227cfd5-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.874 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.874 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.875 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.875 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.875 DEBUG:boto:Method: GET 2023-09-29T22:22:56.875 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.875 DEBUG:boto:Data: 2023-09-29T22:22:56.875 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.875 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.875 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.875 DEBUG:boto:Params: {} 2023-09-29T22:22:56.875 DEBUG:boto:Token: None 2023-09-29T22:22:56.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:56.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0= 2023-09-29T22:22:56.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HYWPpUw0v20/ysgSWSXubANFl/0=', 'Content-Length': '0'} 2023-09-29T22:22:56.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8f0270839682161-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.922 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.922 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.923 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.923 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.923 DEBUG:boto:Method: GET 2023-09-29T22:22:56.923 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.923 DEBUG:boto:Data: 2023-09-29T22:22:56.923 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.923 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.923 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.923 DEBUG:boto:Params: {} 2023-09-29T22:22:56.923 DEBUG:boto:Token: None 2023-09-29T22:22:56.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000382681b6ef82e0ca-0065174e40-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.974 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.975 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:56.975 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:56.975 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.975 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.976 DEBUG:boto:Method: GET 2023-09-29T22:22:56.976 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:56.976 DEBUG:boto:Data: 2023-09-29T22:22:56.976 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.976 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.976 DEBUG:boto:Params: {} 2023-09-29T22:22:56.976 DEBUG:boto:Token: None 2023-09-29T22:22:56.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/ 2023-09-29T22:22:56.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA= 2023-09-29T22:22:56.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gAk8FQP/mn4Ob9v9VhcT0+doWvA=', 'Content-Length': '0'} 2023-09-29T22:22:56.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5da6ab295fc619-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.979 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:56.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.980 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.980 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.980 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.980 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.980 DEBUG:boto:Method: GET 2023-09-29T22:22:56.980 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.981 DEBUG:boto:Data: 2023-09-29T22:22:56.981 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.981 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.981 DEBUG:boto:Params: {} 2023-09-29T22:22:56.981 DEBUG:boto:Token: None 2023-09-29T22:22:56.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.981 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944= 2023-09-29T22:22:56.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944=', 'Content-Length': '0'} 2023-09-29T22:22:56.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0fdfc29a176e33b-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.983 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.984 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.984 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.984 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.984 DEBUG:boto:Method: GET 2023-09-29T22:22:56.984 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.984 DEBUG:boto:Data: 2023-09-29T22:22:56.984 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.984 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.984 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.984 DEBUG:boto:Params: {} 2023-09-29T22:22:56.985 DEBUG:boto:Token: None 2023-09-29T22:22:56.985 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.985 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944= 2023-09-29T22:22:56.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944=', 'Content-Length': '0'} 2023-09-29T22:22:56.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c43da7fe46f14d4-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.987 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.987 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.987 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.987 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.987 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.987 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.987 DEBUG:boto:Data: 2023-09-29T22:22:56.988 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.988 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.988 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.988 DEBUG:boto:Params: {} 2023-09-29T22:22:56.988 DEBUG:boto:Token: None 2023-09-29T22:22:56.988 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJPzloFsm7vrWDsd4g6YxwFYp3E= 2023-09-29T22:22:56.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJPzloFsm7vrWDsd4g6YxwFYp3E=', 'Content-Length': '0'} 2023-09-29T22:22:56.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1a71cb21e00c8fd-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.990 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.990 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.991 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.991 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.991 DEBUG:boto:Method: HEAD 2023-09-29T22:22:56.991 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.991 DEBUG:boto:Data: 2023-09-29T22:22:56.991 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.991 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:56.991 DEBUG:boto:Port: 8000 2023-09-29T22:22:56.991 DEBUG:boto:Params: {} 2023-09-29T22:22:56.991 DEBUG:boto:Token: None 2023-09-29T22:22:56.991 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJPzloFsm7vrWDsd4g6YxwFYp3E= 2023-09-29T22:22:56.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJPzloFsm7vrWDsd4g6YxwFYp3E=', 'Content-Length': '0'} 2023-09-29T22:22:56.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072df2244e30af504-0065174e40-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:56.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:56.994 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:56.994 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:56.994 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.994 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.994 DEBUG:boto:Method: GET 2023-09-29T22:22:56.994 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.994 DEBUG:boto:Data: 2023-09-29T22:22:56.994 DEBUG:boto:Headers: {} 2023-09-29T22:22:56.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:56.995 DEBUG:boto:Port: 8001 2023-09-29T22:22:56.995 DEBUG:boto:Params: {} 2023-09-29T22:22:56.995 DEBUG:boto:Token: None 2023-09-29T22:22:56.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:56 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:56.995 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944= 2023-09-29T22:22:56.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3YIGxEgaEe67QldWxIcAAUqr944=', 'Content-Length': '0'} 2023-09-29T22:22:56.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000f7a63881a53879-0065174e40-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.038 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.039 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.039 DEBUG:boto:path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:57.039 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:57.039 DEBUG:boto:Method: GET 2023-09-29T22:22:57.039 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:57.039 DEBUG:boto:Data: 2023-09-29T22:22:57.039 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.039 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.039 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.040 DEBUG:boto:Params: {} 2023-09-29T22:22:57.040 DEBUG:boto:Token: None 2023-09-29T22:22:57.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl 2023-09-29T22:22:57.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:70FhyoH6OiUs5mR+M5eEl1W4Gjw= 2023-09-29T22:22:57.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:70FhyoH6OiUs5mR+M5eEl1W4Gjw=', 'Content-Length': '0'} 2023-09-29T22:22:57.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'Fdu7slEJ9B4w6NbWPM.cHdXkeB9PhJl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000129dd9aa28033de2-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.042 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.043 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.043 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.043 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.043 DEBUG:boto:Method: GET 2023-09-29T22:22:57.043 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.043 DEBUG:boto:Data: 2023-09-29T22:22:57.043 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.043 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.043 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.043 DEBUG:boto:Params: {} 2023-09-29T22:22:57.043 DEBUG:boto:Token: None 2023-09-29T22:22:57.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a3671d82fff7117-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.086 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.087 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.087 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.087 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.087 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.087 DEBUG:boto:Method: GET 2023-09-29T22:22:57.087 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.087 DEBUG:boto:Data: 2023-09-29T22:22:57.087 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.088 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.088 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.088 DEBUG:boto:Params: {} 2023-09-29T22:22:57.088 DEBUG:boto:Token: None 2023-09-29T22:22:57.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000182be191711d0300-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.091 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.091 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.091 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.092 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.092 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.092 DEBUG:boto:Data: 2023-09-29T22:22:57.092 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.092 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.092 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.092 DEBUG:boto:Params: {} 2023-09-29T22:22:57.092 DEBUG:boto:Token: None 2023-09-29T22:22:57.092 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.092 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003960f4f4b360ebf9-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.095 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.096 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.096 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.096 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.096 DEBUG:boto:Data: 2023-09-29T22:22:57.096 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.096 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.096 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.096 DEBUG:boto:Params: {} 2023-09-29T22:22:57.096 DEBUG:boto:Token: None 2023-09-29T22:22:57.096 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.097 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071a023094a39ba0a-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.100 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.100 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.100 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.101 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.101 DEBUG:boto:Method: GET 2023-09-29T22:22:57.101 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.101 DEBUG:boto:Data: 2023-09-29T22:22:57.101 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.101 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.101 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.101 DEBUG:boto:Params: {} 2023-09-29T22:22:57.101 DEBUG:boto:Token: None 2023-09-29T22:22:57.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.102 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f15ecb094750da4-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.146 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.146 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.147 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.147 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.147 DEBUG:boto:Method: GET 2023-09-29T22:22:57.147 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.147 DEBUG:boto:Data: 2023-09-29T22:22:57.147 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.147 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.147 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.147 DEBUG:boto:Params: {} 2023-09-29T22:22:57.147 DEBUG:boto:Token: None 2023-09-29T22:22:57.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001effc3ccd720e1b4-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.190 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.191 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.191 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.191 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.191 DEBUG:boto:Method: GET 2023-09-29T22:22:57.191 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.191 DEBUG:boto:Data: 2023-09-29T22:22:57.191 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.191 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.191 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.192 DEBUG:boto:Params: {} 2023-09-29T22:22:57.192 DEBUG:boto:Token: None 2023-09-29T22:22:57.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.192 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687cd88f16e979c1-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.238 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.239 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.239 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.239 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.239 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.239 DEBUG:boto:Method: GET 2023-09-29T22:22:57.239 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.239 DEBUG:boto:Data: 2023-09-29T22:22:57.239 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.239 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.239 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.239 DEBUG:boto:Params: {} 2023-09-29T22:22:57.240 DEBUG:boto:Token: None 2023-09-29T22:22:57.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000458d1d5ad0d11998-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.282 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.283 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.283 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.283 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.283 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.283 DEBUG:boto:Method: GET 2023-09-29T22:22:57.283 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.283 DEBUG:boto:Data: 2023-09-29T22:22:57.283 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.284 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.284 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.284 DEBUG:boto:Params: {} 2023-09-29T22:22:57.284 DEBUG:boto:Token: None 2023-09-29T22:22:57.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74= 2023-09-29T22:22:57.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74=', 'Content-Length': '0'} 2023-09-29T22:22:57.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c438df49d2d3ff-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.287 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.287 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.287 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.287 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.287 DEBUG:boto:Method: GET 2023-09-29T22:22:57.287 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.287 DEBUG:boto:Data: 2023-09-29T22:22:57.287 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.287 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.288 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.288 DEBUG:boto:Params: {} 2023-09-29T22:22:57.288 DEBUG:boto:Token: None 2023-09-29T22:22:57.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74= 2023-09-29T22:22:57.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74=', 'Content-Length': '0'} 2023-09-29T22:22:57.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c3c830d008fe425-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.290 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.291 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.291 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.291 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.291 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.291 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.291 DEBUG:boto:Data: 2023-09-29T22:22:57.291 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.291 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.291 DEBUG:boto:Params: {} 2023-09-29T22:22:57.292 DEBUG:boto:Token: None 2023-09-29T22:22:57.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/O0ln4nPHNGDkHt9yP/Y4JLKbBs= 2023-09-29T22:22:57.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/O0ln4nPHNGDkHt9yP/Y4JLKbBs=', 'Content-Length': '0'} 2023-09-29T22:22:57.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033025d96f4940a44-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.294 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.294 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.294 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.294 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.294 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.295 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.295 DEBUG:boto:Data: 2023-09-29T22:22:57.295 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.295 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.295 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.295 DEBUG:boto:Params: {} 2023-09-29T22:22:57.295 DEBUG:boto:Token: None 2023-09-29T22:22:57.295 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/O0ln4nPHNGDkHt9yP/Y4JLKbBs= 2023-09-29T22:22:57.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/O0ln4nPHNGDkHt9yP/Y4JLKbBs=', 'Content-Length': '0'} 2023-09-29T22:22:57.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007560532d386a144b-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.298 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.298 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.298 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.298 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.298 DEBUG:boto:Method: GET 2023-09-29T22:22:57.298 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.299 DEBUG:boto:Data: 2023-09-29T22:22:57.299 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.299 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.299 DEBUG:boto:Params: {} 2023-09-29T22:22:57.299 DEBUG:boto:Token: None 2023-09-29T22:22:57.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74= 2023-09-29T22:22:57.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74=', 'Content-Length': '0'} 2023-09-29T22:22:57.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000303029f693663b2-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.302 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.302 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.302 DEBUG:boto:path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.302 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.302 DEBUG:boto:Method: GET 2023-09-29T22:22:57.302 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.302 DEBUG:boto:Data: 2023-09-29T22:22:57.302 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.303 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.303 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.303 DEBUG:boto:Params: {} 2023-09-29T22:22:57.303 DEBUG:boto:Token: None 2023-09-29T22:22:57.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc 2023-09-29T22:22:57.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74= 2023-09-29T22:22:57.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2RqTl9iUjE81sq4F4QSQc6/er74=', 'Content-Length': '0'} 2023-09-29T22:22:57.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'N0xc2mTHYugKUYrfwv6R6Qf.tHuMvtc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd9213ecac6f2b36-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.346 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.347 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.347 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.347 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.347 DEBUG:boto:Method: GET 2023-09-29T22:22:57.347 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.347 DEBUG:boto:Data: 2023-09-29T22:22:57.347 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.347 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.347 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.347 DEBUG:boto:Params: {} 2023-09-29T22:22:57.348 DEBUG:boto:Token: None 2023-09-29T22:22:57.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.348 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001145bf45af0a2df4-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.351 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.351 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.351 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.351 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.351 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.352 DEBUG:boto:Method: GET 2023-09-29T22:22:57.352 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.352 DEBUG:boto:Data: 2023-09-29T22:22:57.352 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.352 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.352 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.352 DEBUG:boto:Params: {} 2023-09-29T22:22:57.352 DEBUG:boto:Token: None 2023-09-29T22:22:57.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.353 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a9dfcdf2b8537-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.355 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.355 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.356 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.356 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.356 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.356 DEBUG:boto:Data: 2023-09-29T22:22:57.356 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.356 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.356 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.356 DEBUG:boto:Params: {} 2023-09-29T22:22:57.357 DEBUG:boto:Token: None 2023-09-29T22:22:57.357 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071ba46cc321497dd-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.361 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.361 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.361 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.362 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.362 DEBUG:boto:Data: 2023-09-29T22:22:57.362 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.362 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.362 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.362 DEBUG:boto:Params: {} 2023-09-29T22:22:57.362 DEBUG:boto:Token: None 2023-09-29T22:22:57.362 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.363 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000933bc3c92478d5aa-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.366 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.366 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.366 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.366 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.366 DEBUG:boto:Method: GET 2023-09-29T22:22:57.366 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.367 DEBUG:boto:Data: 2023-09-29T22:22:57.367 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.367 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.367 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.367 DEBUG:boto:Params: {} 2023-09-29T22:22:57.367 DEBUG:boto:Token: None 2023-09-29T22:22:57.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.367 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dfb7db4dc0655b9-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.414 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.415 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.415 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.415 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.415 DEBUG:boto:Method: GET 2023-09-29T22:22:57.415 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.415 DEBUG:boto:Data: 2023-09-29T22:22:57.415 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.415 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.415 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.415 DEBUG:boto:Params: {} 2023-09-29T22:22:57.416 DEBUG:boto:Token: None 2023-09-29T22:22:57.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084df1a1312d79653-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.458 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.459 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.459 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.459 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.459 DEBUG:boto:Method: GET 2023-09-29T22:22:57.459 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.459 DEBUG:boto:Data: 2023-09-29T22:22:57.459 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.459 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.460 DEBUG:boto:Params: {} 2023-09-29T22:22:57.460 DEBUG:boto:Token: None 2023-09-29T22:22:57.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f76712b3428682f-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.463 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.463 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.463 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.464 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.464 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.464 DEBUG:boto:Method: GET 2023-09-29T22:22:57.464 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.464 DEBUG:boto:Data: 2023-09-29T22:22:57.464 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.464 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.464 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.464 DEBUG:boto:Params: {} 2023-09-29T22:22:57.465 DEBUG:boto:Token: None 2023-09-29T22:22:57.465 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.465 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0317cc841649cf-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.468 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.468 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.468 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.469 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.469 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.469 DEBUG:boto:Method: GET 2023-09-29T22:22:57.469 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.469 DEBUG:boto:Data: 2023-09-29T22:22:57.469 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.469 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.469 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.469 DEBUG:boto:Params: {} 2023-09-29T22:22:57.469 DEBUG:boto:Token: None 2023-09-29T22:22:57.470 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.470 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E= 2023-09-29T22:22:57.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E=', 'Content-Length': '0'} 2023-09-29T22:22:57.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fcdc27e7c036386-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.472 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.472 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.472 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.472 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.473 DEBUG:boto:Method: GET 2023-09-29T22:22:57.473 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.473 DEBUG:boto:Data: 2023-09-29T22:22:57.473 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.473 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.473 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.473 DEBUG:boto:Params: {} 2023-09-29T22:22:57.473 DEBUG:boto:Token: None 2023-09-29T22:22:57.474 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.474 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E= 2023-09-29T22:22:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E=', 'Content-Length': '0'} 2023-09-29T22:22:57.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b182ec08cea398df-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.518 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.518 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.519 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.519 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.519 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.519 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.519 DEBUG:boto:Data: 2023-09-29T22:22:57.519 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.519 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.519 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.519 DEBUG:boto:Params: {} 2023-09-29T22:22:57.519 DEBUG:boto:Token: None 2023-09-29T22:22:57.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.520 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nJwSzvrRZlSq+RDtVXlAMJBGmjM= 2023-09-29T22:22:57.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nJwSzvrRZlSq+RDtVXlAMJBGmjM=', 'Content-Length': '0'} 2023-09-29T22:22:57.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a01c6114cde3fd3-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.522 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.522 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.522 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.522 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.522 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.522 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.522 DEBUG:boto:Data: 2023-09-29T22:22:57.523 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.523 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.523 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.523 DEBUG:boto:Params: {} 2023-09-29T22:22:57.523 DEBUG:boto:Token: None 2023-09-29T22:22:57.523 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.523 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nJwSzvrRZlSq+RDtVXlAMJBGmjM= 2023-09-29T22:22:57.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nJwSzvrRZlSq+RDtVXlAMJBGmjM=', 'Content-Length': '0'} 2023-09-29T22:22:57.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e984fabc8ce79f0-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.525 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.526 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.526 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.526 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.526 DEBUG:boto:Method: GET 2023-09-29T22:22:57.526 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.526 DEBUG:boto:Data: 2023-09-29T22:22:57.526 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.526 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.526 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.526 DEBUG:boto:Params: {} 2023-09-29T22:22:57.527 DEBUG:boto:Token: None 2023-09-29T22:22:57.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.527 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E= 2023-09-29T22:22:57.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E=', 'Content-Length': '0'} 2023-09-29T22:22:57.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bf55216fdeddac0-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.570 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.570 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.571 DEBUG:boto:path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.571 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.571 DEBUG:boto:Method: GET 2023-09-29T22:22:57.571 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.571 DEBUG:boto:Data: 2023-09-29T22:22:57.571 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.571 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.571 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.571 DEBUG:boto:Params: {} 2023-09-29T22:22:57.571 DEBUG:boto:Token: None 2023-09-29T22:22:57.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC 2023-09-29T22:22:57.572 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E= 2023-09-29T22:22:57.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e4C5KKwdtfSW+/qAjeoa5ZmU39E=', 'Content-Length': '0'} 2023-09-29T22:22:57.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'PTPlqPNXFgb6F3p0fEU-rPyLk8RXduC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dac154362a07291-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.614 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.614 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.615 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.615 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.615 DEBUG:boto:Method: GET 2023-09-29T22:22:57.615 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.615 DEBUG:boto:Data: 2023-09-29T22:22:57.615 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.615 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.615 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.615 DEBUG:boto:Params: {} 2023-09-29T22:22:57.615 DEBUG:boto:Token: None 2023-09-29T22:22:57.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.616 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005278a7ff14d967e6-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.618 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.619 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.619 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.619 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.619 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.619 DEBUG:boto:Method: GET 2023-09-29T22:22:57.619 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.619 DEBUG:boto:Data: 2023-09-29T22:22:57.620 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.620 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.620 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.620 DEBUG:boto:Params: {} 2023-09-29T22:22:57.620 DEBUG:boto:Token: None 2023-09-29T22:22:57.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003b29c8f77bef56c-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.623 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.623 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.623 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.623 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.624 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.624 DEBUG:boto:Data: 2023-09-29T22:22:57.624 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.624 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.624 DEBUG:boto:Params: {} 2023-09-29T22:22:57.624 DEBUG:boto:Token: None 2023-09-29T22:22:57.624 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3e89f598acca8b1-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.628 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.628 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.628 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.628 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.628 DEBUG:boto:Data: 2023-09-29T22:22:57.628 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.628 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.628 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.628 DEBUG:boto:Params: {} 2023-09-29T22:22:57.629 DEBUG:boto:Token: None 2023-09-29T22:22:57.629 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.629 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fafa57fea77af55-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.632 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.633 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.633 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.633 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.633 DEBUG:boto:Method: GET 2023-09-29T22:22:57.633 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.633 DEBUG:boto:Data: 2023-09-29T22:22:57.633 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.633 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.633 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.633 DEBUG:boto:Params: {} 2023-09-29T22:22:57.634 DEBUG:boto:Token: None 2023-09-29T22:22:57.634 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.634 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c1f3af31c3309fa-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.678 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.678 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.678 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.679 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.679 DEBUG:boto:Method: GET 2023-09-29T22:22:57.679 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.679 DEBUG:boto:Data: 2023-09-29T22:22:57.679 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.679 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.679 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.679 DEBUG:boto:Params: {} 2023-09-29T22:22:57.679 DEBUG:boto:Token: None 2023-09-29T22:22:57.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000381e2b7528f66d53-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.722 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.723 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.723 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.723 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.723 DEBUG:boto:Method: GET 2023-09-29T22:22:57.723 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.723 DEBUG:boto:Data: 2023-09-29T22:22:57.723 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.723 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.723 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.724 DEBUG:boto:Params: {} 2023-09-29T22:22:57.724 DEBUG:boto:Token: None 2023-09-29T22:22:57.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01da150921f949b-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.727 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.727 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.727 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.727 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.727 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.727 DEBUG:boto:Method: GET 2023-09-29T22:22:57.728 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.728 DEBUG:boto:Data: 2023-09-29T22:22:57.728 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.728 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.728 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.728 DEBUG:boto:Params: {} 2023-09-29T22:22:57.728 DEBUG:boto:Token: None 2023-09-29T22:22:57.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98e6a89c3b2d989-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.731 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.732 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.732 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.732 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.732 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.732 DEBUG:boto:Method: GET 2023-09-29T22:22:57.732 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.732 DEBUG:boto:Data: 2023-09-29T22:22:57.732 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.732 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.732 DEBUG:boto:Params: {} 2023-09-29T22:22:57.733 DEBUG:boto:Token: None 2023-09-29T22:22:57.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.733 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8= 2023-09-29T22:22:57.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8=', 'Content-Length': '0'} 2023-09-29T22:22:57.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000713aeba3619e8426-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.735 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.735 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.735 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.735 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.736 DEBUG:boto:Method: GET 2023-09-29T22:22:57.736 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.736 DEBUG:boto:Data: 2023-09-29T22:22:57.736 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.736 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.736 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.736 DEBUG:boto:Params: {} 2023-09-29T22:22:57.736 DEBUG:boto:Token: None 2023-09-29T22:22:57.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8= 2023-09-29T22:22:57.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8=', 'Content-Length': '0'} 2023-09-29T22:22:57.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcf4a098cce7863c-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.739 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.739 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.739 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.739 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.739 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.739 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.739 DEBUG:boto:Data: 2023-09-29T22:22:57.739 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.740 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.740 DEBUG:boto:Params: {} 2023-09-29T22:22:57.740 DEBUG:boto:Token: None 2023-09-29T22:22:57.740 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oShzJQZCIbOZm3foSxfG+9JoP2g= 2023-09-29T22:22:57.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oShzJQZCIbOZm3foSxfG+9JoP2g=', 'Content-Length': '0'} 2023-09-29T22:22:57.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004628bc4d202323d1-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.742 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.742 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.742 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.743 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.743 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.743 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.743 DEBUG:boto:Data: 2023-09-29T22:22:57.743 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.743 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.743 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.743 DEBUG:boto:Params: {} 2023-09-29T22:22:57.743 DEBUG:boto:Token: None 2023-09-29T22:22:57.743 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.744 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oShzJQZCIbOZm3foSxfG+9JoP2g= 2023-09-29T22:22:57.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oShzJQZCIbOZm3foSxfG+9JoP2g=', 'Content-Length': '0'} 2023-09-29T22:22:57.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b26a2c47675c275d-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.746 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.746 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.746 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.746 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.746 DEBUG:boto:Method: GET 2023-09-29T22:22:57.746 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.746 DEBUG:boto:Data: 2023-09-29T22:22:57.747 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.747 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.747 DEBUG:boto:Params: {} 2023-09-29T22:22:57.747 DEBUG:boto:Token: None 2023-09-29T22:22:57.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.747 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8= 2023-09-29T22:22:57.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8=', 'Content-Length': '0'} 2023-09-29T22:22:57.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004909149542298394-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.790 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.791 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.791 DEBUG:boto:path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.791 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.791 DEBUG:boto:Method: GET 2023-09-29T22:22:57.791 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.791 DEBUG:boto:Data: 2023-09-29T22:22:57.791 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.791 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.791 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.792 DEBUG:boto:Params: {} 2023-09-29T22:22:57.792 DEBUG:boto:Token: None 2023-09-29T22:22:57.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=5670vOkpV3J9eMlK7bQCZbnI9iPRJox 2023-09-29T22:22:57.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8= 2023-09-29T22:22:57.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IzrjX8o+b6ccPjT5c0/5vjSzF/8=', 'Content-Length': '0'} 2023-09-29T22:22:57.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', '5670vOkpV3J9eMlK7bQCZbnI9iPRJox'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086645f3935c006de-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.794 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.794 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.794 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.794 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.795 DEBUG:boto:Method: GET 2023-09-29T22:22:57.795 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.795 DEBUG:boto:Data: 2023-09-29T22:22:57.795 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.795 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.795 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.795 DEBUG:boto:Params: {} 2023-09-29T22:22:57.795 DEBUG:boto:Token: None 2023-09-29T22:22:57.795 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.795 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce7388854c452b58-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.842 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.843 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.843 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.843 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.843 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.843 DEBUG:boto:Method: GET 2023-09-29T22:22:57.843 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.843 DEBUG:boto:Data: 2023-09-29T22:22:57.843 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.844 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.844 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.844 DEBUG:boto:Params: {} 2023-09-29T22:22:57.844 DEBUG:boto:Token: None 2023-09-29T22:22:57.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.844 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a2a530438b9b67a-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.847 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.847 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.847 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.848 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.848 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.848 DEBUG:boto:Data: 2023-09-29T22:22:57.848 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.848 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.848 DEBUG:boto:Params: {} 2023-09-29T22:22:57.848 DEBUG:boto:Token: None 2023-09-29T22:22:57.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.848 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071ebc66894b090cd-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.852 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.852 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.852 DEBUG:boto:Method: HEAD 2023-09-29T22:22:57.852 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:57.852 DEBUG:boto:Data: 2023-09-29T22:22:57.852 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.853 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.853 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.853 DEBUG:boto:Params: {} 2023-09-29T22:22:57.853 DEBUG:boto:Token: None 2023-09-29T22:22:57.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj 2023-09-29T22:22:57.853 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c= 2023-09-29T22:22:57.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ngrs4r2cjQWv/90MWkxR2g6Ou7c=', 'Content-Length': '0'} 2023-09-29T22:22:57.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7a2d2b1a6e56b21-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:57.856 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.856 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.857 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.857 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.857 DEBUG:boto:Method: GET 2023-09-29T22:22:57.857 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.857 DEBUG:boto:Data: 2023-09-29T22:22:57.857 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.857 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.857 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.857 DEBUG:boto:Params: {} 2023-09-29T22:22:57.857 DEBUG:boto:Token: None 2023-09-29T22:22:57.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.858 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009133fc119306320a-0065174e41-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.902 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:57.902 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:57.902 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.903 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.903 DEBUG:boto:Method: GET 2023-09-29T22:22:57.903 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.903 DEBUG:boto:Data: 2023-09-29T22:22:57.903 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.903 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.903 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.903 DEBUG:boto:Params: {} 2023-09-29T22:22:57.903 DEBUG:boto:Token: None 2023-09-29T22:22:57.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:57.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI= 2023-09-29T22:22:57.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:smvPNoI9uCUkec1x5qCTwkhSeMI=', 'Content-Length': '0'} 2023-09-29T22:22:57.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef826f577ab1bd9a-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.946 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.946 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.947 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.947 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.947 DEBUG:boto:Method: GET 2023-09-29T22:22:57.947 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.947 DEBUG:boto:Data: 2023-09-29T22:22:57.947 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.947 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:57.947 DEBUG:boto:Port: 8001 2023-09-29T22:22:57.947 DEBUG:boto:Params: {} 2023-09-29T22:22:57.947 DEBUG:boto:Token: None 2023-09-29T22:22:57.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c2304b4ad8ed37-0065174e41-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:57.994 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:57.995 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:57.995 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:57.995 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.995 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.995 DEBUG:boto:Method: GET 2023-09-29T22:22:57.995 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:57.996 DEBUG:boto:Data: 2023-09-29T22:22:57.996 DEBUG:boto:Headers: {} 2023-09-29T22:22:57.996 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:57.996 DEBUG:boto:Port: 8000 2023-09-29T22:22:57.996 DEBUG:boto:Params: {} 2023-09-29T22:22:57.996 DEBUG:boto:Token: None 2023-09-29T22:22:57.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:57 GMT /dmuhsu-19/ 2023-09-29T22:22:57.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8= 2023-09-29T22:22:57.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wB73DN53Zf/hAbOtKr835mNfYl8=', 'Content-Length': '0'} 2023-09-29T22:22:57.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14dfc73a8bcd6f1-0065174e41-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.046 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.047 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.047 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.047 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.047 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.047 DEBUG:boto:Method: GET 2023-09-29T22:22:58.047 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.047 DEBUG:boto:Data: 2023-09-29T22:22:58.048 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.048 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.048 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.048 DEBUG:boto:Params: {} 2023-09-29T22:22:58.048 DEBUG:boto:Token: None 2023-09-29T22:22:58.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s= 2023-09-29T22:22:58.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s=', 'Content-Length': '0'} 2023-09-29T22:22:58.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045293fed99a450b0-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.050 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.051 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.051 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.051 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.051 DEBUG:boto:Method: GET 2023-09-29T22:22:58.051 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.051 DEBUG:boto:Data: 2023-09-29T22:22:58.051 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.051 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.051 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.051 DEBUG:boto:Params: {} 2023-09-29T22:22:58.052 DEBUG:boto:Token: None 2023-09-29T22:22:58.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s= 2023-09-29T22:22:58.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s=', 'Content-Length': '0'} 2023-09-29T22:22:58.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eadf0c3407bb212-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.054 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.054 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.054 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.055 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.055 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.055 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.055 DEBUG:boto:Data: 2023-09-29T22:22:58.055 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.055 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.055 DEBUG:boto:Params: {} 2023-09-29T22:22:58.055 DEBUG:boto:Token: None 2023-09-29T22:22:58.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BAWhr3oAFrnnSasJDuXC62x+IwQ= 2023-09-29T22:22:58.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BAWhr3oAFrnnSasJDuXC62x+IwQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052ba420739a2481f-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.058 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.058 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.058 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.058 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.058 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.058 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.058 DEBUG:boto:Data: 2023-09-29T22:22:58.058 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.058 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.059 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.059 DEBUG:boto:Params: {} 2023-09-29T22:22:58.059 DEBUG:boto:Token: None 2023-09-29T22:22:58.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.059 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BAWhr3oAFrnnSasJDuXC62x+IwQ= 2023-09-29T22:22:58.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BAWhr3oAFrnnSasJDuXC62x+IwQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000145383f41caa5707-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.061 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.061 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.061 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.062 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.062 DEBUG:boto:Method: GET 2023-09-29T22:22:58.062 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.062 DEBUG:boto:Data: 2023-09-29T22:22:58.062 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.062 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.062 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.062 DEBUG:boto:Params: {} 2023-09-29T22:22:58.062 DEBUG:boto:Token: None 2023-09-29T22:22:58.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.063 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s= 2023-09-29T22:22:58.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s=', 'Content-Length': '0'} 2023-09-29T22:22:58.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e609a6e0ef331e22-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.065 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.065 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.065 DEBUG:boto:path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.065 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.065 DEBUG:boto:Method: GET 2023-09-29T22:22:58.065 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.065 DEBUG:boto:Data: 2023-09-29T22:22:58.065 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.065 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.066 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.066 DEBUG:boto:Params: {} 2023-09-29T22:22:58.066 DEBUG:boto:Token: None 2023-09-29T22:22:58.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky 2023-09-29T22:22:58.066 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s= 2023-09-29T22:22:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Xgk7q0VkEuNEl+9DaYnhC+wJ0/s=', 'Content-Length': '0'} 2023-09-29T22:22:58.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'KiXedrIktpOkLGXNWQzhwbI2HyTS7Ky'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044a15e2069d0e23d-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.110 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.110 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.111 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.111 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.111 DEBUG:boto:Method: GET 2023-09-29T22:22:58.111 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.111 DEBUG:boto:Data: 2023-09-29T22:22:58.111 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.111 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.111 DEBUG:boto:Params: {} 2023-09-29T22:22:58.111 DEBUG:boto:Token: None 2023-09-29T22:22:58.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e352f81f2e9d86-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.115 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.115 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.115 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.115 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.115 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.116 DEBUG:boto:Method: GET 2023-09-29T22:22:58.116 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.116 DEBUG:boto:Data: 2023-09-29T22:22:58.116 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.116 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.116 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.116 DEBUG:boto:Params: {} 2023-09-29T22:22:58.116 DEBUG:boto:Token: None 2023-09-29T22:22:58.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd305389eaba9323-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.119 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.119 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.120 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.120 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.120 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.120 DEBUG:boto:Data: 2023-09-29T22:22:58.120 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.120 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.120 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.120 DEBUG:boto:Params: {} 2023-09-29T22:22:58.120 DEBUG:boto:Token: None 2023-09-29T22:22:58.121 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000175ace5bb7d91433-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.124 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.124 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.124 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.124 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.125 DEBUG:boto:Data: 2023-09-29T22:22:58.125 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.125 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.125 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.125 DEBUG:boto:Params: {} 2023-09-29T22:22:58.125 DEBUG:boto:Token: None 2023-09-29T22:22:58.125 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.125 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1310ad450baa1de-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.129 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.129 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.129 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.129 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.129 DEBUG:boto:Method: GET 2023-09-29T22:22:58.129 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.129 DEBUG:boto:Data: 2023-09-29T22:22:58.129 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.130 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.130 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.130 DEBUG:boto:Params: {} 2023-09-29T22:22:58.130 DEBUG:boto:Token: None 2023-09-29T22:22:58.130 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.130 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b46c29db80b23234-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.174 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.174 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.174 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.175 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.175 DEBUG:boto:Method: GET 2023-09-29T22:22:58.175 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.175 DEBUG:boto:Data: 2023-09-29T22:22:58.175 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.175 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.175 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.175 DEBUG:boto:Params: {} 2023-09-29T22:22:58.175 DEBUG:boto:Token: None 2023-09-29T22:22:58.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7d21abc666d9dda-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.218 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.218 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.219 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.219 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.219 DEBUG:boto:Method: GET 2023-09-29T22:22:58.219 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.219 DEBUG:boto:Data: 2023-09-29T22:22:58.219 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.219 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.219 DEBUG:boto:Params: {} 2023-09-29T22:22:58.219 DEBUG:boto:Token: None 2023-09-29T22:22:58.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620c22f08d64d7d5-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.222 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.223 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.223 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.223 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.223 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.223 DEBUG:boto:Method: GET 2023-09-29T22:22:58.223 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.223 DEBUG:boto:Data: 2023-09-29T22:22:58.224 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.224 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.224 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.224 DEBUG:boto:Params: {} 2023-09-29T22:22:58.224 DEBUG:boto:Token: None 2023-09-29T22:22:58.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062733e32ee257f52-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.227 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.227 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.228 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.228 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.228 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.228 DEBUG:boto:Method: GET 2023-09-29T22:22:58.228 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.228 DEBUG:boto:Data: 2023-09-29T22:22:58.228 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.228 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.228 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.229 DEBUG:boto:Params: {} 2023-09-29T22:22:58.229 DEBUG:boto:Token: None 2023-09-29T22:22:58.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.229 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0= 2023-09-29T22:22:58.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0=', 'Content-Length': '0'} 2023-09-29T22:22:58.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a206037d6988ac83-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.231 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.231 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.231 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.231 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.232 DEBUG:boto:Method: GET 2023-09-29T22:22:58.232 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.232 DEBUG:boto:Data: 2023-09-29T22:22:58.232 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.232 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.232 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.232 DEBUG:boto:Params: {} 2023-09-29T22:22:58.232 DEBUG:boto:Token: None 2023-09-29T22:22:58.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.233 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0= 2023-09-29T22:22:58.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0=', 'Content-Length': '0'} 2023-09-29T22:22:58.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f2c05e509fc53b-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.278 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.278 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.279 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.279 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.279 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.279 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.279 DEBUG:boto:Data: 2023-09-29T22:22:58.279 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.279 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.279 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.279 DEBUG:boto:Params: {} 2023-09-29T22:22:58.279 DEBUG:boto:Token: None 2023-09-29T22:22:58.280 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.280 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BKOFFl+ud12MgKkfXXFbdBZmUHc= 2023-09-29T22:22:58.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BKOFFl+ud12MgKkfXXFbdBZmUHc=', 'Content-Length': '0'} 2023-09-29T22:22:58.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018165d4be8353653-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.282 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.282 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.282 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.282 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.282 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.282 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.282 DEBUG:boto:Data: 2023-09-29T22:22:58.283 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.283 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.283 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.283 DEBUG:boto:Params: {} 2023-09-29T22:22:58.283 DEBUG:boto:Token: None 2023-09-29T22:22:58.283 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.283 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BKOFFl+ud12MgKkfXXFbdBZmUHc= 2023-09-29T22:22:58.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BKOFFl+ud12MgKkfXXFbdBZmUHc=', 'Content-Length': '0'} 2023-09-29T22:22:58.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000970e999b22d51e4a-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.286 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.286 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.286 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.286 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.286 DEBUG:boto:Method: GET 2023-09-29T22:22:58.286 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.286 DEBUG:boto:Data: 2023-09-29T22:22:58.286 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.286 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.287 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.287 DEBUG:boto:Params: {} 2023-09-29T22:22:58.287 DEBUG:boto:Token: None 2023-09-29T22:22:58.287 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.287 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0= 2023-09-29T22:22:58.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0=', 'Content-Length': '0'} 2023-09-29T22:22:58.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce36a115103bb314-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.330 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.330 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.330 DEBUG:boto:path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.331 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.331 DEBUG:boto:Method: GET 2023-09-29T22:22:58.331 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.331 DEBUG:boto:Data: 2023-09-29T22:22:58.331 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.331 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.331 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.331 DEBUG:boto:Params: {} 2023-09-29T22:22:58.331 DEBUG:boto:Token: None 2023-09-29T22:22:58.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=P47GybDIgug9dC.sEibuF-MGXAmUiua 2023-09-29T22:22:58.332 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0= 2023-09-29T22:22:58.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dpFj2bBueMuDBZ9RBHrG+xtw/F0=', 'Content-Length': '0'} 2023-09-29T22:22:58.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'P47GybDIgug9dC.sEibuF-MGXAmUiua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000313d208903523867-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.374 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.374 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.374 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.375 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.375 DEBUG:boto:Method: GET 2023-09-29T22:22:58.375 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.375 DEBUG:boto:Data: 2023-09-29T22:22:58.375 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.375 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.375 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.375 DEBUG:boto:Params: {} 2023-09-29T22:22:58.375 DEBUG:boto:Token: None 2023-09-29T22:22:58.375 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.376 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2aad22d4ef1723b-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.378 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.379 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.379 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.379 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.379 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.379 DEBUG:boto:Method: GET 2023-09-29T22:22:58.379 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.379 DEBUG:boto:Data: 2023-09-29T22:22:58.379 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.379 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.380 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.380 DEBUG:boto:Params: {} 2023-09-29T22:22:58.380 DEBUG:boto:Token: None 2023-09-29T22:22:58.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0873094f4f3b074-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.383 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.383 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.383 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.384 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.384 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.385 DEBUG:boto:Data: 2023-09-29T22:22:58.385 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.385 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.385 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.385 DEBUG:boto:Params: {} 2023-09-29T22:22:58.385 DEBUG:boto:Token: None 2023-09-29T22:22:58.385 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.385 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047624095e974bd2d-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.389 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.389 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.389 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.389 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.389 DEBUG:boto:Data: 2023-09-29T22:22:58.389 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.389 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.389 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.389 DEBUG:boto:Params: {} 2023-09-29T22:22:58.389 DEBUG:boto:Token: None 2023-09-29T22:22:58.390 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.390 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c8e8f8774a99bac-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.393 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.393 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.393 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.393 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.394 DEBUG:boto:Method: GET 2023-09-29T22:22:58.394 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.394 DEBUG:boto:Data: 2023-09-29T22:22:58.394 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.394 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.394 DEBUG:boto:Params: {} 2023-09-29T22:22:58.394 DEBUG:boto:Token: None 2023-09-29T22:22:58.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb5802a6d0cf58b1-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.438 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.439 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.439 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.439 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.439 DEBUG:boto:Method: GET 2023-09-29T22:22:58.439 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.439 DEBUG:boto:Data: 2023-09-29T22:22:58.439 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.439 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.439 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.440 DEBUG:boto:Params: {} 2023-09-29T22:22:58.440 DEBUG:boto:Token: None 2023-09-29T22:22:58.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0587d4e81b6ca63-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.486 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.486 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.486 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.487 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.487 DEBUG:boto:Method: GET 2023-09-29T22:22:58.487 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.487 DEBUG:boto:Data: 2023-09-29T22:22:58.487 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.487 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.487 DEBUG:boto:Params: {} 2023-09-29T22:22:58.487 DEBUG:boto:Token: None 2023-09-29T22:22:58.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.488 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000958d7cd321c872ef-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.491 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.491 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.491 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.491 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.491 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.491 DEBUG:boto:Method: GET 2023-09-29T22:22:58.492 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.492 DEBUG:boto:Data: 2023-09-29T22:22:58.492 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.492 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.492 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.492 DEBUG:boto:Params: {} 2023-09-29T22:22:58.492 DEBUG:boto:Token: None 2023-09-29T22:22:58.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cb37fe6e656ef55-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.538 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.539 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.539 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.539 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.539 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.539 DEBUG:boto:Method: GET 2023-09-29T22:22:58.539 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.539 DEBUG:boto:Data: 2023-09-29T22:22:58.540 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.540 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.540 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.540 DEBUG:boto:Params: {} 2023-09-29T22:22:58.540 DEBUG:boto:Token: None 2023-09-29T22:22:58.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.540 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4= 2023-09-29T22:22:58.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4=', 'Content-Length': '0'} 2023-09-29T22:22:58.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c7a89ff300b6a48-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.542 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.543 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.543 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.543 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.543 DEBUG:boto:Method: GET 2023-09-29T22:22:58.543 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.543 DEBUG:boto:Data: 2023-09-29T22:22:58.543 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.543 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.543 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.543 DEBUG:boto:Params: {} 2023-09-29T22:22:58.544 DEBUG:boto:Token: None 2023-09-29T22:22:58.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.544 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4= 2023-09-29T22:22:58.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4=', 'Content-Length': '0'} 2023-09-29T22:22:58.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf8af1abf7d7b161-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.546 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.546 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.546 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.547 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.547 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.547 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.547 DEBUG:boto:Data: 2023-09-29T22:22:58.547 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.547 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.547 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.547 DEBUG:boto:Params: {} 2023-09-29T22:22:58.547 DEBUG:boto:Token: None 2023-09-29T22:22:58.547 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.548 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PR1h+06Pykej8p5yqmiDYl6gTTk= 2023-09-29T22:22:58.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PR1h+06Pykej8p5yqmiDYl6gTTk=', 'Content-Length': '0'} 2023-09-29T22:22:58.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007353cf55b954af66-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.550 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.550 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.550 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.550 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.550 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.550 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.550 DEBUG:boto:Data: 2023-09-29T22:22:58.551 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.551 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.551 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.551 DEBUG:boto:Params: {} 2023-09-29T22:22:58.551 DEBUG:boto:Token: None 2023-09-29T22:22:58.551 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.551 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PR1h+06Pykej8p5yqmiDYl6gTTk= 2023-09-29T22:22:58.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PR1h+06Pykej8p5yqmiDYl6gTTk=', 'Content-Length': '0'} 2023-09-29T22:22:58.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059fd83453d75e57e-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.553 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.554 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.554 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.554 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.554 DEBUG:boto:Method: GET 2023-09-29T22:22:58.554 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.554 DEBUG:boto:Data: 2023-09-29T22:22:58.554 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.554 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.554 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.555 DEBUG:boto:Params: {} 2023-09-29T22:22:58.555 DEBUG:boto:Token: None 2023-09-29T22:22:58.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.555 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4= 2023-09-29T22:22:58.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4=', 'Content-Length': '0'} 2023-09-29T22:22:58.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035e9b5b3ad8a5ca1-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.557 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.557 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.557 DEBUG:boto:path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.557 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.557 DEBUG:boto:Method: GET 2023-09-29T22:22:58.558 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.558 DEBUG:boto:Data: 2023-09-29T22:22:58.558 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.558 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.558 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.558 DEBUG:boto:Params: {} 2023-09-29T22:22:58.558 DEBUG:boto:Token: None 2023-09-29T22:22:58.558 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=SScUnRPCrcSNC5QQL1tehNEyL0eXHiM 2023-09-29T22:22:58.558 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4= 2023-09-29T22:22:58.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GAt48S4yDNrF+JLb0NPTtYigku4=', 'Content-Length': '0'} 2023-09-29T22:22:58.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'SScUnRPCrcSNC5QQL1tehNEyL0eXHiM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aceeea593efe074c-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.602 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.603 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.603 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.603 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.603 DEBUG:boto:Method: GET 2023-09-29T22:22:58.603 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.603 DEBUG:boto:Data: 2023-09-29T22:22:58.603 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.603 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.604 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.604 DEBUG:boto:Params: {} 2023-09-29T22:22:58.604 DEBUG:boto:Token: None 2023-09-29T22:22:58.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.604 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af730680939663cc-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.607 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.608 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.608 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.608 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.608 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.608 DEBUG:boto:Method: GET 2023-09-29T22:22:58.608 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.608 DEBUG:boto:Data: 2023-09-29T22:22:58.608 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.608 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.608 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.609 DEBUG:boto:Params: {} 2023-09-29T22:22:58.609 DEBUG:boto:Token: None 2023-09-29T22:22:58.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.609 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c01b9e83aa2778e-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.611 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.612 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.612 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.612 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.612 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.612 DEBUG:boto:Data: 2023-09-29T22:22:58.612 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.612 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.612 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.613 DEBUG:boto:Params: {} 2023-09-29T22:22:58.613 DEBUG:boto:Token: None 2023-09-29T22:22:58.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d901188dc9e53220-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.616 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.617 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.617 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.617 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.617 DEBUG:boto:Data: 2023-09-29T22:22:58.617 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.617 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.617 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.617 DEBUG:boto:Params: {} 2023-09-29T22:22:58.617 DEBUG:boto:Token: None 2023-09-29T22:22:58.617 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.618 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c320de51b94750b2-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.621 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.621 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.621 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.622 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.622 DEBUG:boto:Method: GET 2023-09-29T22:22:58.622 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.622 DEBUG:boto:Data: 2023-09-29T22:22:58.622 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.622 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.622 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.622 DEBUG:boto:Params: {} 2023-09-29T22:22:58.622 DEBUG:boto:Token: None 2023-09-29T22:22:58.623 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.623 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052eb557a64c5c08f-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.666 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.666 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.667 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.667 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.667 DEBUG:boto:Method: GET 2023-09-29T22:22:58.667 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.667 DEBUG:boto:Data: 2023-09-29T22:22:58.667 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.667 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.667 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.667 DEBUG:boto:Params: {} 2023-09-29T22:22:58.667 DEBUG:boto:Token: None 2023-09-29T22:22:58.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b35a7d1727e121-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.710 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.710 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.711 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.711 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.711 DEBUG:boto:Method: GET 2023-09-29T22:22:58.711 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.711 DEBUG:boto:Data: 2023-09-29T22:22:58.711 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.711 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.711 DEBUG:boto:Params: {} 2023-09-29T22:22:58.712 DEBUG:boto:Token: None 2023-09-29T22:22:58.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.712 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed8959a572976020-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.714 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.715 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.715 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.715 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.715 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.715 DEBUG:boto:Method: GET 2023-09-29T22:22:58.715 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.715 DEBUG:boto:Data: 2023-09-29T22:22:58.715 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.716 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.716 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.716 DEBUG:boto:Params: {} 2023-09-29T22:22:58.716 DEBUG:boto:Token: None 2023-09-29T22:22:58.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.716 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000995c7e0f2940c7d5-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.719 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.719 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.719 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.719 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.719 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.720 DEBUG:boto:Method: GET 2023-09-29T22:22:58.720 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.720 DEBUG:boto:Data: 2023-09-29T22:22:58.720 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.720 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.720 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.720 DEBUG:boto:Params: {} 2023-09-29T22:22:58.720 DEBUG:boto:Token: None 2023-09-29T22:22:58.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.721 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI= 2023-09-29T22:22:58.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI=', 'Content-Length': '0'} 2023-09-29T22:22:58.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006edd5397cdc2abe6-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.723 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.723 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.723 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.723 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.723 DEBUG:boto:Method: GET 2023-09-29T22:22:58.723 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.723 DEBUG:boto:Data: 2023-09-29T22:22:58.724 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.724 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.724 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.724 DEBUG:boto:Params: {} 2023-09-29T22:22:58.724 DEBUG:boto:Token: None 2023-09-29T22:22:58.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI= 2023-09-29T22:22:58.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI=', 'Content-Length': '0'} 2023-09-29T22:22:58.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003224e7f0615bd3c0-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.766 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.766 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.767 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.767 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.767 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.767 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.767 DEBUG:boto:Data: 2023-09-29T22:22:58.767 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.767 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.767 DEBUG:boto:Params: {} 2023-09-29T22:22:58.767 DEBUG:boto:Token: None 2023-09-29T22:22:58.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UNqhCz/R6zDjeoWARCtcx2V9aZI= 2023-09-29T22:22:58.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UNqhCz/R6zDjeoWARCtcx2V9aZI=', 'Content-Length': '0'} 2023-09-29T22:22:58.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbdd945bb0fbcb14-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.770 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.770 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.770 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.770 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.770 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.770 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.770 DEBUG:boto:Data: 2023-09-29T22:22:58.771 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.771 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.771 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.771 DEBUG:boto:Params: {} 2023-09-29T22:22:58.771 DEBUG:boto:Token: None 2023-09-29T22:22:58.771 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UNqhCz/R6zDjeoWARCtcx2V9aZI= 2023-09-29T22:22:58.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UNqhCz/R6zDjeoWARCtcx2V9aZI=', 'Content-Length': '0'} 2023-09-29T22:22:58.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006230c1cb554492a3-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.773 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.774 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.774 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.774 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.774 DEBUG:boto:Method: GET 2023-09-29T22:22:58.774 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.774 DEBUG:boto:Data: 2023-09-29T22:22:58.774 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.774 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.774 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.774 DEBUG:boto:Params: {} 2023-09-29T22:22:58.775 DEBUG:boto:Token: None 2023-09-29T22:22:58.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.775 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI= 2023-09-29T22:22:58.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI=', 'Content-Length': '0'} 2023-09-29T22:22:58.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980a87ed569c3aa6-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.818 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.819 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.819 DEBUG:boto:path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.819 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.819 DEBUG:boto:Method: GET 2023-09-29T22:22:58.819 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.819 DEBUG:boto:Data: 2023-09-29T22:22:58.819 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.819 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.819 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.819 DEBUG:boto:Params: {} 2023-09-29T22:22:58.820 DEBUG:boto:Token: None 2023-09-29T22:22:58.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=t6DBuD82EJPXFqtJn.Sul7SXH3eEr9w 2023-09-29T22:22:58.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI= 2023-09-29T22:22:58.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CuypeBIR1FSoHH0mimOhmWn9rnI=', 'Content-Length': '0'} 2023-09-29T22:22:58.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 't6DBuD82EJPXFqtJn.Sul7SXH3eEr9w'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059544860f367f143-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.862 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.863 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.863 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.863 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.863 DEBUG:boto:Method: GET 2023-09-29T22:22:58.863 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.863 DEBUG:boto:Data: 2023-09-29T22:22:58.863 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.863 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.863 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.863 DEBUG:boto:Params: {} 2023-09-29T22:22:58.864 DEBUG:boto:Token: None 2023-09-29T22:22:58.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ae06163b6cb5ec-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.866 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.867 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.867 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.867 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.867 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.867 DEBUG:boto:Method: GET 2023-09-29T22:22:58.867 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.867 DEBUG:boto:Data: 2023-09-29T22:22:58.868 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.868 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.868 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.868 DEBUG:boto:Params: {} 2023-09-29T22:22:58.868 DEBUG:boto:Token: None 2023-09-29T22:22:58.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.868 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba5f7704b9b37c0-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.871 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.871 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.871 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.871 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.871 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.871 DEBUG:boto:Data: 2023-09-29T22:22:58.872 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.872 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.872 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.872 DEBUG:boto:Params: {} 2023-09-29T22:22:58.872 DEBUG:boto:Token: None 2023-09-29T22:22:58.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005eaab19cb63c996-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.875 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.875 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.875 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.876 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:58.876 DEBUG:boto:Data: 2023-09-29T22:22:58.876 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.876 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.876 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.876 DEBUG:boto:Params: {} 2023-09-29T22:22:58.876 DEBUG:boto:Token: None 2023-09-29T22:22:58.876 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj 2023-09-29T22:22:58.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8= 2023-09-29T22:22:58.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZWcTC2x0onK+bos1P/tTxpZdbT8=', 'Content-Length': '0'} 2023-09-29T22:22:58.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b77d46ea25840da-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.880 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.880 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.880 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.880 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.880 DEBUG:boto:Method: GET 2023-09-29T22:22:58.880 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.880 DEBUG:boto:Data: 2023-09-29T22:22:58.880 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.881 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.881 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.881 DEBUG:boto:Params: {} 2023-09-29T22:22:58.881 DEBUG:boto:Token: None 2023-09-29T22:22:58.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.881 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071034edd5c51e0b1-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.926 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.926 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.927 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.927 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.927 DEBUG:boto:Method: GET 2023-09-29T22:22:58.927 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.927 DEBUG:boto:Data: 2023-09-29T22:22:58.927 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.927 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.927 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.927 DEBUG:boto:Params: {} 2023-09-29T22:22:58.927 DEBUG:boto:Token: None 2023-09-29T22:22:58.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:58.928 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ= 2023-09-29T22:22:58.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:m9IOPqRhrqisyuLxCJqnxwbIBGQ=', 'Content-Length': '0'} 2023-09-29T22:22:58.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc48a9959bcc9c46-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.970 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.971 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.971 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.971 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.971 DEBUG:boto:Method: GET 2023-09-29T22:22:58.971 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.971 DEBUG:boto:Data: 2023-09-29T22:22:58.971 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.971 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.971 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.971 DEBUG:boto:Params: {} 2023-09-29T22:22:58.972 DEBUG:boto:Token: None 2023-09-29T22:22:58.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000caa3914446a0a6b-0065174e42-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.975 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.975 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:58.975 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:58.975 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.975 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.976 DEBUG:boto:Method: GET 2023-09-29T22:22:58.976 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:58.976 DEBUG:boto:Data: 2023-09-29T22:22:58.976 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.976 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.976 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.976 DEBUG:boto:Params: {} 2023-09-29T22:22:58.976 DEBUG:boto:Token: None 2023-09-29T22:22:58.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/ 2023-09-29T22:22:58.977 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU= 2023-09-29T22:22:58.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yDH5C5NTWv1V3rpa1N4kEEslpaU=', 'Content-Length': '0'} 2023-09-29T22:22:58.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc85dae2fdcd518-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.979 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:58.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.980 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.980 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.980 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.980 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.980 DEBUG:boto:Method: GET 2023-09-29T22:22:58.980 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.980 DEBUG:boto:Data: 2023-09-29T22:22:58.981 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.981 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.981 DEBUG:boto:Params: {} 2023-09-29T22:22:58.981 DEBUG:boto:Token: None 2023-09-29T22:22:58.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.981 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI= 2023-09-29T22:22:58.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI=', 'Content-Length': '0'} 2023-09-29T22:22:58.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6568602db99c29f-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.983 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.984 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.984 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.984 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.984 DEBUG:boto:Method: GET 2023-09-29T22:22:58.984 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.984 DEBUG:boto:Data: 2023-09-29T22:22:58.984 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.984 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.984 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.985 DEBUG:boto:Params: {} 2023-09-29T22:22:58.985 DEBUG:boto:Token: None 2023-09-29T22:22:58.985 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.985 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI= 2023-09-29T22:22:58.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI=', 'Content-Length': '0'} 2023-09-29T22:22:58.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac5c06b26a0a0515-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.987 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.987 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.987 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.987 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.988 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.988 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.988 DEBUG:boto:Data: 2023-09-29T22:22:58.988 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.988 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.988 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.988 DEBUG:boto:Params: {} 2023-09-29T22:22:58.988 DEBUG:boto:Token: None 2023-09-29T22:22:58.988 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.989 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/feAUbc33cWagid0HJiqOQxF22A= 2023-09-29T22:22:58.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/feAUbc33cWagid0HJiqOQxF22A=', 'Content-Length': '0'} 2023-09-29T22:22:58.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001c18e288db214a8-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.991 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.991 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.991 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.991 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.991 DEBUG:boto:Method: HEAD 2023-09-29T22:22:58.991 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.991 DEBUG:boto:Data: 2023-09-29T22:22:58.991 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.991 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:58.992 DEBUG:boto:Port: 8000 2023-09-29T22:22:58.992 DEBUG:boto:Params: {} 2023-09-29T22:22:58.992 DEBUG:boto:Token: None 2023-09-29T22:22:58.992 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/feAUbc33cWagid0HJiqOQxF22A= 2023-09-29T22:22:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/feAUbc33cWagid0HJiqOQxF22A=', 'Content-Length': '0'} 2023-09-29T22:22:58.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002be9efa2e7405e75-0065174e42-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:58.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:58.994 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:58.994 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:58.994 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.994 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.995 DEBUG:boto:Method: GET 2023-09-29T22:22:58.995 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.995 DEBUG:boto:Data: 2023-09-29T22:22:58.995 DEBUG:boto:Headers: {} 2023-09-29T22:22:58.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:58.995 DEBUG:boto:Port: 8001 2023-09-29T22:22:58.995 DEBUG:boto:Params: {} 2023-09-29T22:22:58.995 DEBUG:boto:Token: None 2023-09-29T22:22:58.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:58 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:58.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI= 2023-09-29T22:22:58.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:njWTNC7AisGBpAolSiGrew/xvPI=', 'Content-Length': '0'} 2023-09-29T22:22:58.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008cb4f6f946b4f66-0065174e42-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.042 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:59.042 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:59.042 DEBUG:boto:path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:59.043 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:59.043 DEBUG:boto:Method: GET 2023-09-29T22:22:59.043 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:59.043 DEBUG:boto:Data: 2023-09-29T22:22:59.043 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.043 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:59.043 DEBUG:boto:Port: 8000 2023-09-29T22:22:59.043 DEBUG:boto:Params: {} 2023-09-29T22:22:59.043 DEBUG:boto:Token: None 2023-09-29T22:22:59.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/obj?versionId=zNfF-8Qv4tifWWzD8LieUTqh64w4TOk 2023-09-29T22:22:59.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cqJQ07r7Xa1uuFVVKfSnEGusDvU= 2023-09-29T22:22:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cqJQ07r7Xa1uuFVVKfSnEGusDvU=', 'Content-Length': '0'} 2023-09-29T22:22:59.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'zNfF-8Qv4tifWWzD8LieUTqh64w4TOk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d25f6c93a4386d4-0065174e43-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.046 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:59.046 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:59.046 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.047 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.047 DEBUG:boto:Method: GET 2023-09-29T22:22:59.047 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.047 DEBUG:boto:Data: 2023-09-29T22:22:59.047 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:59.047 DEBUG:boto:Port: 8001 2023-09-29T22:22:59.047 DEBUG:boto:Params: {} 2023-09-29T22:22:59.047 DEBUG:boto:Token: None 2023-09-29T22:22:59.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/ 2023-09-29T22:22:59.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY= 2023-09-29T22:22:59.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY=', 'Content-Length': '0'} 2023-09-29T22:22:59.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51ccda11790a118-0065174e43-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.051 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:59.051 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:59.051 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:59.051 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.051 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.052 DEBUG:boto:Method: GET 2023-09-29T22:22:59.052 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.052 DEBUG:boto:Data: 2023-09-29T22:22:59.052 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.052 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:59.052 DEBUG:boto:Port: 8000 2023-09-29T22:22:59.052 DEBUG:boto:Params: {} 2023-09-29T22:22:59.052 DEBUG:boto:Token: None 2023-09-29T22:22:59.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/ 2023-09-29T22:22:59.053 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY= 2023-09-29T22:22:59.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY=', 'Content-Length': '0'} 2023-09-29T22:22:59.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030088b3dc1b9ae7d-0065174e43-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.055 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:59.056 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:59.056 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:59.056 DEBUG:boto:Method: HEAD 2023-09-29T22:22:59.056 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:59.056 DEBUG:boto:Data: 2023-09-29T22:22:59.056 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.056 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:59.056 DEBUG:boto:Port: 8001 2023-09-29T22:22:59.056 DEBUG:boto:Params: {} 2023-09-29T22:22:59.057 DEBUG:boto:Token: None 2023-09-29T22:22:59.057 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/obj 2023-09-29T22:22:59.057 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7Z5ne2hucVcBRH+q30PQoH2AgrE= 2023-09-29T22:22:59.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7Z5ne2hucVcBRH+q30PQoH2AgrE=', 'Content-Length': '0'} 2023-09-29T22:22:59.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dd10d48f83c8f1c-0065174e43-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.060 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:59.060 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:59.060 DEBUG:boto:Method: HEAD 2023-09-29T22:22:59.061 DEBUG:boto:Path: /dmuhsu-19/obj 2023-09-29T22:22:59.061 DEBUG:boto:Data: 2023-09-29T22:22:59.061 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.061 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:59.061 DEBUG:boto:Port: 8000 2023-09-29T22:22:59.061 DEBUG:boto:Params: {} 2023-09-29T22:22:59.061 DEBUG:boto:Token: None 2023-09-29T22:22:59.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/obj 2023-09-29T22:22:59.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7Z5ne2hucVcBRH+q30PQoH2AgrE= 2023-09-29T22:22:59.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7Z5ne2hucVcBRH+q30PQoH2AgrE=', 'Content-Length': '0'} 2023-09-29T22:22:59.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091132f09409fd599-0065174e43-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:22:59.065 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:59.065 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:59.065 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.065 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.065 DEBUG:boto:Method: GET 2023-09-29T22:22:59.065 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.065 DEBUG:boto:Data: 2023-09-29T22:22:59.065 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.066 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:59.066 DEBUG:boto:Port: 8001 2023-09-29T22:22:59.066 DEBUG:boto:Params: {} 2023-09-29T22:22:59.066 DEBUG:boto:Token: None 2023-09-29T22:22:59.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.066 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y+Ail2nUJ/qjjzsViZ9lvpb+LNQ= 2023-09-29T22:22:59.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y+Ail2nUJ/qjjzsViZ9lvpb+LNQ=', 'Content-Length': '0'} 2023-09-29T22:22:59.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1bbe59fc6d4a7df-0065174e43-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.110 DEBUG:boto:path=/dmuhsu-19/obj 2023-09-29T22:22:59.110 DEBUG:boto:auth_path=/dmuhsu-19/obj 2023-09-29T22:22:59.111 DEBUG:boto:path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.111 DEBUG:boto:auth_path=/dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.111 DEBUG:boto:Method: GET 2023-09-29T22:22:59.111 DEBUG:boto:Path: /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.111 DEBUG:boto:Data: 2023-09-29T22:22:59.111 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.111 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:59.111 DEBUG:boto:Port: 8000 2023-09-29T22:22:59.111 DEBUG:boto:Params: {} 2023-09-29T22:22:59.112 DEBUG:boto:Token: None 2023-09-29T22:22:59.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/obj?versionId=lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh 2023-09-29T22:22:59.112 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y+Ail2nUJ/qjjzsViZ9lvpb+LNQ= 2023-09-29T22:22:59.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y+Ail2nUJ/qjjzsViZ9lvpb+LNQ=', 'Content-Length': '0'} 2023-09-29T22:22:59.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:19:59 GMT'), ('x-amz-version-id', 'lr-1f0YGIF5FSQ7y.7AlfCOIl94EQsh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1ac7ff80164d40-0065174e43-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.154 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:59.155 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:59.155 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.155 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.155 DEBUG:boto:Method: GET 2023-09-29T22:22:59.155 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.155 DEBUG:boto:Data: 2023-09-29T22:22:59.155 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.155 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:22:59.156 DEBUG:boto:Port: 8001 2023-09-29T22:22:59.156 DEBUG:boto:Params: {} 2023-09-29T22:22:59.156 DEBUG:boto:Token: None 2023-09-29T22:22:59.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/ 2023-09-29T22:22:59.156 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY= 2023-09-29T22:22:59.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY=', 'Content-Length': '0'} 2023-09-29T22:22:59.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6fa7a4a50fee609-0065174e43-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.159 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:59.160 DEBUG:boto:path=/dmuhsu-19/ 2023-09-29T22:22:59.160 DEBUG:boto:auth_path=/dmuhsu-19/ 2023-09-29T22:22:59.160 DEBUG:boto:path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.160 DEBUG:boto:auth_path=/dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.160 DEBUG:boto:Method: GET 2023-09-29T22:22:59.160 DEBUG:boto:Path: /dmuhsu-19/?prefix=obj 2023-09-29T22:22:59.161 DEBUG:boto:Data: 2023-09-29T22:22:59.161 DEBUG:boto:Headers: {} 2023-09-29T22:22:59.161 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:22:59.161 DEBUG:boto:Port: 8000 2023-09-29T22:22:59.161 DEBUG:boto:Params: {} 2023-09-29T22:22:59.161 DEBUG:boto:Token: None 2023-09-29T22:22:59.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:22:59 GMT /dmuhsu-19/ 2023-09-29T22:22:59.161 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY= 2023-09-29T22:22:59.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:22:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u8zrW4RRNLnHM2vakWtyzdrgVCY=', 'Content-Length': '0'} 2023-09-29T22:22:59.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1fbe122f70712ef-0065174e43-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:22:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:22:59.210 DEBUG:boto:b'dmuhsu-19obj1000falseobj2023-09-29T22:19:59.807Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:22:59.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-19 zones={test-zone3, test-zone2} 2023-09-29T22:22:59.211 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2023-09-29T22:22:59.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.212 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.263 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.264 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.312 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:22:37.516996Z" 2023-09-29T22:22:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.314 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:22:59.315 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.315 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-29T22:22:59.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.316 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.370 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.370 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.417 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:22:59.417 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:22:59.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:22:59.419 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:22:37.516996Z" 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.420 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:22:59Z" 2023-09-29T22:22:59.423 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:22:59.424 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:22:37.516996Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:22:59Z" } 2023-09-29T22:22:59.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:22:37.516996Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:22:59.424 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.425 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.425 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-29T22:22:59.425 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2023-09-29T22:22:59.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.425 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-09-29T22:22:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:22:37.516996Z" 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.516 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:22:59.517 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.517 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2023-09-29T22:22:59.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.518 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.565 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.565 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.609 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 12860396553369717831 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.610 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.611 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001640", 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:22:37.516996Z" 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.612 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:22:59.613 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.614 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:22:59Z" 2023-09-29T22:22:59.615 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:22:59.616 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 12860396553369717831 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:22:37.516996Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:22:59Z" } 2023-09-29T22:22:59.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:22:37.516996Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:22:59.617 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.617 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001640', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:22:59.617 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2023-09-29T22:22:59.617 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2023-09-29T22:22:59.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.617 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.666 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:17:42.523805Z" 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:22:59.714 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:21:15.314892Z" 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004883", 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:20:34.160426Z" 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.715 INFO:teuthology.orchestra.run.smithi184.stdout:] 2023-09-29T22:22:59.716 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:22:59.716 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2023-09-29T22:22:59.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.717 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.764 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.764 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.806 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:22:59.806 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:22:59.806 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:22:59.806 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:22:59.806 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.807 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:17:42.523805Z" 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:22:59.808 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:21:15.314892Z" 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.809 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000004883", 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:20:34.160426Z" 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:22:59.810 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:22:59.811 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:22:59Z" 2023-09-29T22:22:59.812 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:22:59.813 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:42.523805Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:15.314892Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004883", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.160426Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:22:59Z" } 2023-09-29T22:22:59.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:42.523805Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:15.314892Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004883', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.160426Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:22:59.813 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:22:59.813 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:22:59.814 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2023-09-29T22:22:59.814 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2023-09-29T22:22:59.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.814 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.861 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.861 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:17:42.523805Z" 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:21:15.314892Z" 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004883", 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-29T22:20:34.160426Z" 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2023-09-29T22:22:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:22:59.907 INFO:teuthology.orchestra.run.smithi184.stdout:] 2023-09-29T22:22:59.907 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:22:59.908 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2023-09-29T22:22:59.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:22:59.908 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:22:59.955 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:22:59.955 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 13130857650295640028 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:22:59.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:17:42.523805Z" 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.998 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:21:15.314892Z" 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000004883", 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:22:59.999 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:20:34.160426Z" 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.000 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.001 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:22:59Z" 2023-09-29T22:23:00.002 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:23:00.002 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13130857650295640028 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:42.523805Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:15.314892Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004883", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.160426Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:22:59Z" } 2023-09-29T22:23:00.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:42.523805Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:15.314892Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004883', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.160426Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:23:00.003 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:23:00.003 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004883', 3: ''} 2023-09-29T22:23:00.003 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2023-09-29T22:23:00.003 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2023-09-29T22:23:00.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.004 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.047 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.047 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.092 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:23:00.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.092 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:17:00.231645Z" 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:21:20.503940Z" 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:19:56.238845Z" 2023-09-29T22:23:00.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.094 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:23:00.094 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.094 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:23:00.095 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.095 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2023-09-29T22:23:00.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.095 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.143 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.143 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.184 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:23:00.184 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:23:00.184 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 9944456043159136811 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.185 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:17:00.231645Z" 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.186 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:21:20.503940Z" 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:19:56.238845Z" 2023-09-29T22:23:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.189 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:23:00Z" 2023-09-29T22:23:00.190 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:23:00.191 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9944456043159136811 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:00.231645Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:20.503940Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004601", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:19:56.238845Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:23:00Z" } 2023-09-29T22:23:00.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:00.231645Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:20.503940Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:19:56.238845Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:23:00.192 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.192 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.192 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2023-09-29T22:23:00.192 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2023-09-29T22:23:00.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.193 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.236 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.236 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:17:00.231645Z" 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.283 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:21:20.503940Z" 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:19:56.238845Z" 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.285 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:23:00.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.285 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:23:00.286 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.286 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2023-09-29T22:23:00.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.287 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.377 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:23:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:23:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:23:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:23:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 14751276066808193184 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:17:00.231645Z" 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:23:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001358", 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:21:20.503940Z" 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:19:56.238845Z" 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:23:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:23:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:23:00.384 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:23:00.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.384 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:23:00Z" 2023-09-29T22:23:00.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:23:00.385 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14751276066808193184 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:00.231645Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:20.503940Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004601", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:19:56.238845Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:23:00Z" } 2023-09-29T22:23:00.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:00.231645Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:20.503940Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:19:56.238845Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:23:00.385 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.385 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000004601', 3: ''} 2023-09-29T22:23:00.386 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2023-09-29T22:23:00.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-09-29T22:23:00.388 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:00.388 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:00.388 DEBUG:boto:Method: PUT 2023-09-29T22:23:00.388 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:00.388 DEBUG:boto:Data: 2023-09-29T22:23:00.388 DEBUG:boto:Headers: {} 2023-09-29T22:23:00.389 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:00.389 DEBUG:boto:Port: 8000 2023-09-29T22:23:00.389 DEBUG:boto:Params: {} 2023-09-29T22:23:00.389 DEBUG:boto:Token: None 2023-09-29T22:23:00.389 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:23:00 GMT /dmuhsu-20/ 2023-09-29T22:23:00.389 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aChR4e44ep/5ci6NiPYXpp14Bcs= 2023-09-29T22:23:00.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aChR4e44ep/5ci6NiPYXpp14Bcs=', 'Content-Length': '0'} 2023-09-29T22:23:00.402 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a5f299751cbef96-0065174e44-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:00.402 DEBUG:rgw_multi.tests:created bucket=dmuhsu-20 2023-09-29T22:23:00.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.403 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.448 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.449 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.484 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:23:00.498 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:23:00.498 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.498 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:00.498 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:23:00.498 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026180.401687_128.1", 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:23:00.401687Z" 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.500 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:23:00.501 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026180.401687_128.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:00.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:23:00.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.502 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:00.580+0000 7fe551f23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:00.581 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:00.581+0000 7fe551f23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:00.592 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:23:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:23:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:23:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026180.401687_128.1", 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:23:00.401687Z", 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:23:00Z" 2023-09-29T22:23:00.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:23:00.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026180.401687_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:00.401687Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:00Z" } 2023-09-29T22:23:00.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026180.401687_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:00.401687Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:23:00.602 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026180.401687_128.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:00.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026180.401687_128.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:00.602 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:23:00.602 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:23:00.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:00.603 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:00.648 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.648 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:00.675+0000 7efd77b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:00.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:00.676+0000 7efd77b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:23:00.687 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.688 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.689 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:00.690 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026180.401687_128.1", 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.691 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:23:00.401687Z", 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:00.692 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:23:00.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:23:00Z" 2023-09-29T22:23:00.694 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:23:00.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026180.401687_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:00.401687Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:00Z" } 2023-09-29T22:23:00.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026180.401687_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:00.401687Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:23:00.696 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026180.401687_128.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:00.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026180.401687_128.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:00.696 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:23:00.696 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:00.696 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:00.696 DEBUG:boto:Method: HEAD 2023-09-29T22:23:00.696 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:00.696 DEBUG:boto:Data: 2023-09-29T22:23:00.696 DEBUG:boto:Headers: {} 2023-09-29T22:23:00.697 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:00.697 DEBUG:boto:Port: 8000 2023-09-29T22:23:00.697 DEBUG:boto:Params: {} 2023-09-29T22:23:00.697 DEBUG:boto:Token: None 2023-09-29T22:23:00.697 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:00 GMT /dmuhsu-20/ 2023-09-29T22:23:00.697 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q5ynaeeC2Un/mVcO3sLO6Onfur4= 2023-09-29T22:23:00.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q5ynaeeC2Un/mVcO3sLO6Onfur4=', 'Content-Length': '0'} 2023-09-29T22:23:00.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f25e890aef7808ec-0065174e44-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:00.700 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:00.700 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:00.700 DEBUG:boto:Method: PUT 2023-09-29T22:23:00.700 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:00.700 DEBUG:boto:Data: 2023-09-29T22:23:00.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:23:00.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:00.701 DEBUG:boto:Port: 8000 2023-09-29T22:23:00.701 DEBUG:boto:Params: {} 2023-09-29T22:23:00.701 DEBUG:boto:Token: None 2023-09-29T22:23:00.701 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:23:00 GMT /dmuhsu-20/obj 2023-09-29T22:23:00.701 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qp/L2pIYvqS7kWvgLDUkftFVYRY= 2023-09-29T22:23:00.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qp/L2pIYvqS7kWvgLDUkftFVYRY='} 2023-09-29T22:23:00.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008edc2fd27820e86d-0065174e44-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:00.707 DEBUG:rgw_multi.tests:created initial version id=None 2023-09-29T22:23:00.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:23:00.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:00.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:00.707 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:00.759 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:00.759 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:00.793 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:00.793+0000 7f0188244880 1 waiting to reach incremental sync.. 2023-09-29T22:23:05.793 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:05.793+0000 7f0188244880 1 waiting to reach incremental sync.. 2023-09-29T22:23:10.794 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:10.794+0000 7f0188244880 1 waiting to reach incremental sync.. 2023-09-29T22:23:15.796 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:15.795+0000 7f0188244880 1 bucket sync caught up with source: 2023-09-29T22:23:15.796 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:15.796 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:15.796 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:15.795+0000 7f0188244880 0 bucket checkpoint complete 2023-09-29T22:23:15.812 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:23:15.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:15.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:15.813 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:15.872 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:15.907 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:15.908+0000 7f2f35c15880 1 waiting to reach incremental sync.. 2023-09-29T22:23:20.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:20.909+0000 7f2f35c15880 1 bucket sync caught up with source: 2023-09-29T22:23:20.909 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:20.909 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:20.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:20.909+0000 7f2f35c15880 0 bucket checkpoint complete 2023-09-29T22:23:20.925 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:23:20.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:20.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:20.925 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:20.983 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:20.983 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:21.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:21.019+0000 7f02aa611880 1 bucket sync caught up with source: 2023-09-29T22:23:21.018 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.20.6, , ] 2023-09-29T22:23:21.019 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.20.6, , ] 2023-09-29T22:23:21.019 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:21.019+0000 7f02aa611880 0 bucket checkpoint complete 2023-09-29T22:23:21.031 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:23:21.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:21.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:21.032 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:21.083 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:21.083 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:21.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:21.118+0000 7ff13de12880 1 bucket sync caught up with source: 2023-09-29T22:23:21.118 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.20.6, , ] 2023-09-29T22:23:21.118 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.20.6, , ] 2023-09-29T22:23:21.118 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:21.118+0000 7ff13de12880 0 bucket checkpoint complete 2023-09-29T22:23:21.130 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:23:21.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:21.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:21.130 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:21.182 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:21.182 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:21.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:21.214+0000 7fcaeef29880 1 waiting to reach incremental sync.. 2023-09-29T22:23:26.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:26.214+0000 7fcaeef29880 1 waiting to reach incremental sync.. 2023-09-29T22:23:31.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:31.215+0000 7fcaeef29880 1 waiting to reach incremental sync.. 2023-09-29T22:23:36.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:36.215+0000 7fcaeef29880 1 waiting to reach incremental sync.. 2023-09-29T22:23:38.679 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:23:38.678+0000 7f303f796640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:23:41.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:41.216+0000 7fcaeef29880 1 bucket sync caught up with source: 2023-09-29T22:23:41.217 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:23:41.217 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:23:41.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:41.216+0000 7fcaeef29880 0 bucket checkpoint complete 2023-09-29T22:23:41.233 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:23:41.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:41.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:41.233 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:41.295 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:41.295 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:41.334 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:41.334+0000 7f3549136880 1 bucket sync caught up with source: 2023-09-29T22:23:41.334 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:23:41.334 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:23:41.334 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:41.334+0000 7f3549136880 0 bucket checkpoint complete 2023-09-29T22:23:41.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:23:41.347 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.347 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.347 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.348 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.348 DEBUG:boto:Data: 2023-09-29T22:23:41.348 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.348 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.348 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.348 DEBUG:boto:Params: {} 2023-09-29T22:23:41.348 DEBUG:boto:Token: None 2023-09-29T22:23:41.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.349 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000089b2f6ee621cd90-0065174e6d-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.351 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.351 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.351 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.351 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.351 DEBUG:boto:Data: 2023-09-29T22:23:41.351 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.352 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.352 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.352 DEBUG:boto:Params: {} 2023-09-29T22:23:41.352 DEBUG:boto:Token: None 2023-09-29T22:23:41.352 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.352 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3fcba77ba7bc270-0065174e6d-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.355 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:23:41.355 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.355 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.355 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.355 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.355 DEBUG:boto:Method: GET 2023-09-29T22:23:41.355 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.355 DEBUG:boto:Data: 2023-09-29T22:23:41.355 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.356 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.356 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.356 DEBUG:boto:Params: {} 2023-09-29T22:23:41.356 DEBUG:boto:Token: None 2023-09-29T22:23:41.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.356 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c11fb9020edad729-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.359 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:23:41.360 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.360 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.360 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.360 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.360 DEBUG:boto:Method: GET 2023-09-29T22:23:41.360 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.360 DEBUG:boto:Data: 2023-09-29T22:23:41.360 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.361 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.361 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.361 DEBUG:boto:Params: {} 2023-09-29T22:23:41.361 DEBUG:boto:Token: None 2023-09-29T22:23:41.361 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.361 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868c70bbc24ade40-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.364 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.364 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.365 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.365 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.365 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.365 DEBUG:boto:Method: GET 2023-09-29T22:23:41.365 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.365 DEBUG:boto:Data: 2023-09-29T22:23:41.365 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.365 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.365 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.365 DEBUG:boto:Params: {} 2023-09-29T22:23:41.366 DEBUG:boto:Token: None 2023-09-29T22:23:41.366 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.366 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e7b6ea13965cc5-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.368 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.369 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.369 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.369 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.369 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.369 DEBUG:boto:Method: GET 2023-09-29T22:23:41.369 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.369 DEBUG:boto:Data: 2023-09-29T22:23:41.370 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.370 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.370 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.370 DEBUG:boto:Params: {} 2023-09-29T22:23:41.370 DEBUG:boto:Token: None 2023-09-29T22:23:41.370 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.370 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6bdb74887a470cd-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.414 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.415 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.415 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.415 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.415 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.415 DEBUG:boto:Method: GET 2023-09-29T22:23:41.416 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.416 DEBUG:boto:Data: 2023-09-29T22:23:41.416 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.416 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.416 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.416 DEBUG:boto:Params: {} 2023-09-29T22:23:41.416 DEBUG:boto:Token: None 2023-09-29T22:23:41.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffaf5ca9a51f9f8b-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.419 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.419 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.419 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.419 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.419 DEBUG:boto:Method: GET 2023-09-29T22:23:41.419 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.419 DEBUG:boto:Data: 2023-09-29T22:23:41.420 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.420 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.420 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.420 DEBUG:boto:Params: {} 2023-09-29T22:23:41.420 DEBUG:boto:Token: None 2023-09-29T22:23:41.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5f35aed9fe298f9-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.422 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.422 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.423 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.423 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.423 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.423 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.423 DEBUG:boto:Data: 2023-09-29T22:23:41.423 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.423 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.423 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.423 DEBUG:boto:Params: {} 2023-09-29T22:23:41.423 DEBUG:boto:Token: None 2023-09-29T22:23:41.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df177b2400d079ba-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.425 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.425 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.426 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.426 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.426 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.426 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.426 DEBUG:boto:Data: 2023-09-29T22:23:41.426 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.426 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.426 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.427 DEBUG:boto:Params: {} 2023-09-29T22:23:41.427 DEBUG:boto:Token: None 2023-09-29T22:23:41.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.427 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f70ebad8f3e2dd47-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.429 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.430 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.430 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.430 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.430 DEBUG:boto:Method: GET 2023-09-29T22:23:41.430 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.430 DEBUG:boto:Data: 2023-09-29T22:23:41.430 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.430 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.431 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.431 DEBUG:boto:Params: {} 2023-09-29T22:23:41.431 DEBUG:boto:Token: None 2023-09-29T22:23:41.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.431 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000064b5f45e38a62b1-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.433 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.433 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.433 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.433 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.434 DEBUG:boto:Method: GET 2023-09-29T22:23:41.434 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.434 DEBUG:boto:Data: 2023-09-29T22:23:41.434 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.434 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.434 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.434 DEBUG:boto:Params: {} 2023-09-29T22:23:41.435 DEBUG:boto:Token: None 2023-09-29T22:23:41.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.435 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b8f3e0fe42bd732-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.437 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.437 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.437 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.437 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.437 DEBUG:boto:Method: GET 2023-09-29T22:23:41.438 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.438 DEBUG:boto:Data: 2023-09-29T22:23:41.438 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.438 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.438 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.438 DEBUG:boto:Params: {} 2023-09-29T22:23:41.438 DEBUG:boto:Token: None 2023-09-29T22:23:41.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.438 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7178bb9f9515ac2-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.482 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.483 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.483 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.483 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.483 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.483 DEBUG:boto:Method: GET 2023-09-29T22:23:41.483 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.483 DEBUG:boto:Data: 2023-09-29T22:23:41.483 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.484 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.484 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.484 DEBUG:boto:Params: {} 2023-09-29T22:23:41.484 DEBUG:boto:Token: None 2023-09-29T22:23:41.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055436e17a6d53758-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.487 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.487 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.487 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.488 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.488 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.488 DEBUG:boto:Data: 2023-09-29T22:23:41.488 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.488 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.488 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.488 DEBUG:boto:Params: {} 2023-09-29T22:23:41.488 DEBUG:boto:Token: None 2023-09-29T22:23:41.489 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.489 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035e50c2c497104f1-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.491 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.491 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.491 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.491 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.491 DEBUG:boto:Data: 2023-09-29T22:23:41.491 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.491 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.491 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.491 DEBUG:boto:Params: {} 2023-09-29T22:23:41.492 DEBUG:boto:Token: None 2023-09-29T22:23:41.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.492 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b06586b8f14eef1-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.494 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.494 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.494 DEBUG:boto:Method: GET 2023-09-29T22:23:41.494 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.494 DEBUG:boto:Data: 2023-09-29T22:23:41.495 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.495 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.495 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.495 DEBUG:boto:Params: {} 2023-09-29T22:23:41.495 DEBUG:boto:Token: None 2023-09-29T22:23:41.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.495 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b3ab80aeb28385a-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.497 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.497 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.497 DEBUG:boto:Method: GET 2023-09-29T22:23:41.497 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.497 DEBUG:boto:Data: 2023-09-29T22:23:41.497 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.498 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.498 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.498 DEBUG:boto:Params: {} 2023-09-29T22:23:41.498 DEBUG:boto:Token: None 2023-09-29T22:23:41.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.498 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b28d99441263fef-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.500 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.500 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.500 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.501 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.501 DEBUG:boto:Method: GET 2023-09-29T22:23:41.501 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.501 DEBUG:boto:Data: 2023-09-29T22:23:41.501 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.501 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.501 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.501 DEBUG:boto:Params: {} 2023-09-29T22:23:41.501 DEBUG:boto:Token: None 2023-09-29T22:23:41.502 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.502 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e31cef0d78027a-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.504 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.504 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.505 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.505 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.505 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.505 DEBUG:boto:Method: GET 2023-09-29T22:23:41.505 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.505 DEBUG:boto:Data: 2023-09-29T22:23:41.505 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.506 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.506 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.506 DEBUG:boto:Params: {} 2023-09-29T22:23:41.506 DEBUG:boto:Token: None 2023-09-29T22:23:41.506 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.506 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d9049f281a5449-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.509 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:23:41.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:23:41.509 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.509 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.510 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.510 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.510 DEBUG:boto:Data: 2023-09-29T22:23:41.510 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.510 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.510 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.510 DEBUG:boto:Params: {} 2023-09-29T22:23:41.511 DEBUG:boto:Token: None 2023-09-29T22:23:41.511 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.511 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf66585b32cf2e77-0065174e6d-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.514 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.514 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.514 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.514 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.514 DEBUG:boto:Data: 2023-09-29T22:23:41.514 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.514 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.514 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.515 DEBUG:boto:Params: {} 2023-09-29T22:23:41.515 DEBUG:boto:Token: None 2023-09-29T22:23:41.515 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.515 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c727cc45ef0c3b01-0065174e6d-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.517 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:23:41.517 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.517 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.518 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.518 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.518 DEBUG:boto:Method: GET 2023-09-29T22:23:41.518 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.518 DEBUG:boto:Data: 2023-09-29T22:23:41.518 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.518 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.519 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.519 DEBUG:boto:Params: {} 2023-09-29T22:23:41.519 DEBUG:boto:Token: None 2023-09-29T22:23:41.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.519 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a87c1e00b73da7a7-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.522 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.522 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:23:41.523 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.523 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.523 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.523 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.523 DEBUG:boto:Method: GET 2023-09-29T22:23:41.523 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.523 DEBUG:boto:Data: 2023-09-29T22:23:41.523 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.523 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.524 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.524 DEBUG:boto:Params: {} 2023-09-29T22:23:41.524 DEBUG:boto:Token: None 2023-09-29T22:23:41.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.524 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a828437d364f828-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.527 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.527 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.528 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.528 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.528 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.528 DEBUG:boto:Method: GET 2023-09-29T22:23:41.528 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.528 DEBUG:boto:Data: 2023-09-29T22:23:41.528 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.528 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.528 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.528 DEBUG:boto:Params: {} 2023-09-29T22:23:41.529 DEBUG:boto:Token: None 2023-09-29T22:23:41.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.529 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95746937c5b508f-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.531 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.532 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.532 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.532 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.532 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.532 DEBUG:boto:Method: GET 2023-09-29T22:23:41.532 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.532 DEBUG:boto:Data: 2023-09-29T22:23:41.533 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.533 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.533 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.533 DEBUG:boto:Params: {} 2023-09-29T22:23:41.533 DEBUG:boto:Token: None 2023-09-29T22:23:41.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.533 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582776392ce8fa24-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.578 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.579 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.579 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.579 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.579 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.579 DEBUG:boto:Method: GET 2023-09-29T22:23:41.580 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.580 DEBUG:boto:Data: 2023-09-29T22:23:41.580 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.580 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.580 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.580 DEBUG:boto:Params: {} 2023-09-29T22:23:41.580 DEBUG:boto:Token: None 2023-09-29T22:23:41.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.581 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084294e86572197d6-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.583 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.583 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.583 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.583 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.583 DEBUG:boto:Method: GET 2023-09-29T22:23:41.583 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.584 DEBUG:boto:Data: 2023-09-29T22:23:41.584 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.584 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.584 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.584 DEBUG:boto:Params: {} 2023-09-29T22:23:41.584 DEBUG:boto:Token: None 2023-09-29T22:23:41.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b35143a06b3f182-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.586 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.586 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.586 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.587 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.587 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.587 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.587 DEBUG:boto:Data: 2023-09-29T22:23:41.587 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.587 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.587 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.587 DEBUG:boto:Params: {} 2023-09-29T22:23:41.588 DEBUG:boto:Token: None 2023-09-29T22:23:41.588 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c14a7876fb2f0e27-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.590 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.590 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.590 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.590 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.591 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.591 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.591 DEBUG:boto:Data: 2023-09-29T22:23:41.591 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.591 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.591 DEBUG:boto:Params: {} 2023-09-29T22:23:41.591 DEBUG:boto:Token: None 2023-09-29T22:23:41.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000742b7ec85674fedc-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.594 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.594 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.594 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.594 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.594 DEBUG:boto:Method: GET 2023-09-29T22:23:41.594 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.594 DEBUG:boto:Data: 2023-09-29T22:23:41.594 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.594 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.594 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.595 DEBUG:boto:Params: {} 2023-09-29T22:23:41.595 DEBUG:boto:Token: None 2023-09-29T22:23:41.595 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.595 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006519d76272b159eb-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.597 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.597 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.597 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.597 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.598 DEBUG:boto:Method: GET 2023-09-29T22:23:41.598 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.598 DEBUG:boto:Data: 2023-09-29T22:23:41.598 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.598 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.598 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.598 DEBUG:boto:Params: {} 2023-09-29T22:23:41.598 DEBUG:boto:Token: None 2023-09-29T22:23:41.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.599 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b76aeb82bb748a7-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.601 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.601 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.601 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.601 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.601 DEBUG:boto:Method: GET 2023-09-29T22:23:41.601 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.601 DEBUG:boto:Data: 2023-09-29T22:23:41.601 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.601 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.601 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.602 DEBUG:boto:Params: {} 2023-09-29T22:23:41.602 DEBUG:boto:Token: None 2023-09-29T22:23:41.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c1c90789e2fba67-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.605 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.605 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.605 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.605 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.605 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.605 DEBUG:boto:Method: GET 2023-09-29T22:23:41.606 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.606 DEBUG:boto:Data: 2023-09-29T22:23:41.606 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.606 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.606 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.606 DEBUG:boto:Params: {} 2023-09-29T22:23:41.606 DEBUG:boto:Token: None 2023-09-29T22:23:41.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d3465cd81443de-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.609 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.609 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.609 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.609 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.609 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.609 DEBUG:boto:Data: 2023-09-29T22:23:41.610 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.610 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.610 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.610 DEBUG:boto:Params: {} 2023-09-29T22:23:41.610 DEBUG:boto:Token: None 2023-09-29T22:23:41.610 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d961fa3ef96402ad-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.612 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.612 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.612 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.613 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.613 DEBUG:boto:Data: 2023-09-29T22:23:41.613 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.613 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.613 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.613 DEBUG:boto:Params: {} 2023-09-29T22:23:41.613 DEBUG:boto:Token: None 2023-09-29T22:23:41.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c0e7670e2ca5dae-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.615 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.616 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.616 DEBUG:boto:Method: GET 2023-09-29T22:23:41.616 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.616 DEBUG:boto:Data: 2023-09-29T22:23:41.616 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.616 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.616 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.616 DEBUG:boto:Params: {} 2023-09-29T22:23:41.616 DEBUG:boto:Token: None 2023-09-29T22:23:41.617 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.617 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3bd250c7ea584ca-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.619 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.619 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.619 DEBUG:boto:Method: GET 2023-09-29T22:23:41.619 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.619 DEBUG:boto:Data: 2023-09-29T22:23:41.619 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.619 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.619 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.620 DEBUG:boto:Params: {} 2023-09-29T22:23:41.620 DEBUG:boto:Token: None 2023-09-29T22:23:41.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.620 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ae9e33596e0bd47-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.622 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.622 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.622 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.622 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.623 DEBUG:boto:Method: GET 2023-09-29T22:23:41.623 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.623 DEBUG:boto:Data: 2023-09-29T22:23:41.623 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.623 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.623 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.623 DEBUG:boto:Params: {} 2023-09-29T22:23:41.623 DEBUG:boto:Token: None 2023-09-29T22:23:41.623 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.624 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ea7d8bd607f80e-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.626 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.626 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.626 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.626 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.627 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.627 DEBUG:boto:Method: GET 2023-09-29T22:23:41.627 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.627 DEBUG:boto:Data: 2023-09-29T22:23:41.627 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.627 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.627 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.627 DEBUG:boto:Params: {} 2023-09-29T22:23:41.627 DEBUG:boto:Token: None 2023-09-29T22:23:41.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.628 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000939a0e9f0005f9e7-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.630 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:23:41.631 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:23:41.631 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.631 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.631 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.631 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.631 DEBUG:boto:Data: 2023-09-29T22:23:41.631 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.631 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.631 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.631 DEBUG:boto:Params: {} 2023-09-29T22:23:41.632 DEBUG:boto:Token: None 2023-09-29T22:23:41.632 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.632 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010642fa18155bba5-0065174e6d-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.634 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.634 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.634 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.634 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:41.635 DEBUG:boto:Data: 2023-09-29T22:23:41.635 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.635 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.635 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.635 DEBUG:boto:Params: {} 2023-09-29T22:23:41.635 DEBUG:boto:Token: None 2023-09-29T22:23:41.635 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.635 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660= 2023-09-29T22:23:41.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nPBlImNzaEG4TGg5T7LBSXyB660=', 'Content-Length': '0'} 2023-09-29T22:23:41.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afbb692bb3006a84-0065174e6d-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.637 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:23:41.638 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.638 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.638 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.638 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.638 DEBUG:boto:Method: GET 2023-09-29T22:23:41.638 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.638 DEBUG:boto:Data: 2023-09-29T22:23:41.638 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.638 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.638 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.639 DEBUG:boto:Params: {} 2023-09-29T22:23:41.639 DEBUG:boto:Token: None 2023-09-29T22:23:41.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.639 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbd6d837bcf5816-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.641 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:23:41.642 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.642 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.642 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.642 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.642 DEBUG:boto:Method: GET 2023-09-29T22:23:41.642 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.643 DEBUG:boto:Data: 2023-09-29T22:23:41.643 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.643 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.643 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.643 DEBUG:boto:Params: {} 2023-09-29T22:23:41.643 DEBUG:boto:Token: None 2023-09-29T22:23:41.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.643 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0bf37b4d0ca29bf-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.646 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:23:41.646 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.646 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.646 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.646 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.647 DEBUG:boto:Method: GET 2023-09-29T22:23:41.647 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.647 DEBUG:boto:Data: 2023-09-29T22:23:41.647 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.647 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.647 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.647 DEBUG:boto:Params: {} 2023-09-29T22:23:41.647 DEBUG:boto:Token: None 2023-09-29T22:23:41.647 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.648 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ba66258827125a-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.650 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.650 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.651 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.651 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.651 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.651 DEBUG:boto:Method: GET 2023-09-29T22:23:41.651 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:23:41.651 DEBUG:boto:Data: 2023-09-29T22:23:41.651 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.651 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.651 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.651 DEBUG:boto:Params: {} 2023-09-29T22:23:41.652 DEBUG:boto:Token: None 2023-09-29T22:23:41.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versions 2023-09-29T22:23:41.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug= 2023-09-29T22:23:41.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vbhXOGAPp1gViZBbbaZ4IxfbCug=', 'Content-Length': '0'} 2023-09-29T22:23:41.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855285c7c4521676-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.654 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.655 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.655 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.655 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.655 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.655 DEBUG:boto:Method: GET 2023-09-29T22:23:41.655 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.655 DEBUG:boto:Data: 2023-09-29T22:23:41.656 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.656 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.656 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.656 DEBUG:boto:Params: {} 2023-09-29T22:23:41.656 DEBUG:boto:Token: None 2023-09-29T22:23:41.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d46e6b1be8d2bba-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.658 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.658 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.658 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.659 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.659 DEBUG:boto:Method: GET 2023-09-29T22:23:41.659 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.659 DEBUG:boto:Data: 2023-09-29T22:23:41.659 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.659 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.659 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.659 DEBUG:boto:Params: {} 2023-09-29T22:23:41.659 DEBUG:boto:Token: None 2023-09-29T22:23:41.659 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.660 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1e22de5e8e15e42-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.662 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.662 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.662 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.662 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.662 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.662 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.662 DEBUG:boto:Data: 2023-09-29T22:23:41.662 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.662 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.663 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.663 DEBUG:boto:Params: {} 2023-09-29T22:23:41.663 DEBUG:boto:Token: None 2023-09-29T22:23:41.663 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.663 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb0d7584750992db-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.665 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.665 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.665 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.665 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.665 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.665 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.666 DEBUG:boto:Data: 2023-09-29T22:23:41.666 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.666 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.666 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.666 DEBUG:boto:Params: {} 2023-09-29T22:23:41.666 DEBUG:boto:Token: None 2023-09-29T22:23:41.666 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.666 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc= 2023-09-29T22:23:41.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:viWAWXXsaXP97gN+Qzpr4FZVgkc=', 'Content-Length': '0'} 2023-09-29T22:23:41.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080dbdf82c2f0b2f0-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.669 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.669 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.669 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.669 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.669 DEBUG:boto:Method: GET 2023-09-29T22:23:41.669 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.669 DEBUG:boto:Data: 2023-09-29T22:23:41.670 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.670 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.670 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.670 DEBUG:boto:Params: {} 2023-09-29T22:23:41.670 DEBUG:boto:Token: None 2023-09-29T22:23:41.670 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.670 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047fb76327c2d5287-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.672 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.672 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.673 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.673 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.673 DEBUG:boto:Method: GET 2023-09-29T22:23:41.673 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.673 DEBUG:boto:Data: 2023-09-29T22:23:41.673 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.673 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.673 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.673 DEBUG:boto:Params: {} 2023-09-29T22:23:41.674 DEBUG:boto:Token: None 2023-09-29T22:23:41.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:23:41.674 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo= 2023-09-29T22:23:41.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:11Om42ndUABg61DlX8XbfnUyPRo=', 'Content-Length': '0'} 2023-09-29T22:23:41.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005490fef4a4622d83-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.676 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.676 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.677 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.677 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.677 DEBUG:boto:Method: GET 2023-09-29T22:23:41.677 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.677 DEBUG:boto:Data: 2023-09-29T22:23:41.677 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.677 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.677 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.678 DEBUG:boto:Params: {} 2023-09-29T22:23:41.678 DEBUG:boto:Token: None 2023-09-29T22:23:41.678 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.678 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a7df3921c0ec9dc-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.681 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.681 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.681 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.681 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.681 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.682 DEBUG:boto:Method: GET 2023-09-29T22:23:41.682 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.682 DEBUG:boto:Data: 2023-09-29T22:23:41.682 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.682 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.682 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.682 DEBUG:boto:Params: {} 2023-09-29T22:23:41.682 DEBUG:boto:Token: None 2023-09-29T22:23:41.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.683 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac070d89560bb5a-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.726 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.727 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.727 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.727 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.727 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.728 DEBUG:boto:Data: 2023-09-29T22:23:41.728 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.728 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.728 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.728 DEBUG:boto:Params: {} 2023-09-29T22:23:41.728 DEBUG:boto:Token: None 2023-09-29T22:23:41.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb076fc9fe5e0f22-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.731 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.731 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.731 DEBUG:boto:Method: HEAD 2023-09-29T22:23:41.731 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.731 DEBUG:boto:Data: 2023-09-29T22:23:41.731 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.731 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.731 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.731 DEBUG:boto:Params: {} 2023-09-29T22:23:41.732 DEBUG:boto:Token: None 2023-09-29T22:23:41.732 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU= 2023-09-29T22:23:41.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WccZcIZPuqGPGOlHUhf8q5lg4XU=', 'Content-Length': '0'} 2023-09-29T22:23:41.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006dfd11924b26c4e9-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:23:41.734 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.734 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.734 DEBUG:boto:Method: GET 2023-09-29T22:23:41.735 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.735 DEBUG:boto:Data: 2023-09-29T22:23:41.735 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.735 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.735 DEBUG:boto:Params: {} 2023-09-29T22:23:41.735 DEBUG:boto:Token: None 2023-09-29T22:23:41.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.735 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092d41febfda9605e-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.737 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:41.737 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:41.737 DEBUG:boto:Method: GET 2023-09-29T22:23:41.738 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:41.738 DEBUG:boto:Data: 2023-09-29T22:23:41.738 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.738 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.738 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.738 DEBUG:boto:Params: {} 2023-09-29T22:23:41.738 DEBUG:boto:Token: None 2023-09-29T22:23:41.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/obj 2023-09-29T22:23:41.739 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw= 2023-09-29T22:23:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OM4bpMHHjqSMfsPegWjTCKXsBXw=', 'Content-Length': '0'} 2023-09-29T22:23:41.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d220a0986f9fdd9b-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.740 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.741 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.741 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.741 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.741 DEBUG:boto:Method: GET 2023-09-29T22:23:41.741 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.741 DEBUG:boto:Data: 2023-09-29T22:23:41.741 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.741 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:23:41.741 DEBUG:boto:Port: 8001 2023-09-29T22:23:41.741 DEBUG:boto:Params: {} 2023-09-29T22:23:41.742 DEBUG:boto:Token: None 2023-09-29T22:23:41.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.742 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c888ed2708f2e075-0065174e6d-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.745 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.745 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.745 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.745 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.745 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.745 DEBUG:boto:Method: GET 2023-09-29T22:23:41.746 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:23:41.746 DEBUG:boto:Data: 2023-09-29T22:23:41.746 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.746 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:23:41.746 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.746 DEBUG:boto:Params: {} 2023-09-29T22:23:41.746 DEBUG:boto:Token: None 2023-09-29T22:23:41.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/ 2023-09-29T22:23:41.746 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ= 2023-09-29T22:23:41.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SeOdMk7erzmML6hSS5PPdXTrbuQ=', 'Content-Length': '0'} 2023-09-29T22:23:41.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5dc1e42b537261-0065174e6d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.749 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:41.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:23:41.749 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:41.749 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:41.750 DEBUG:boto:path=/dmuhsu-20/?versioning 2023-09-29T22:23:41.750 DEBUG:boto:auth_path=/dmuhsu-20/?versioning 2023-09-29T22:23:41.750 DEBUG:boto:Method: PUT 2023-09-29T22:23:41.750 DEBUG:boto:Path: /dmuhsu-20/?versioning 2023-09-29T22:23:41.750 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:23:41.750 DEBUG:boto:Headers: {} 2023-09-29T22:23:41.750 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:41.750 DEBUG:boto:Port: 8000 2023-09-29T22:23:41.750 DEBUG:boto:Params: {} 2023-09-29T22:23:41.751 DEBUG:boto:Token: None 2023-09-29T22:23:41.751 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:23:41 GMT /dmuhsu-20/?versioning 2023-09-29T22:23:41.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J03bPWZgI4AY2VFDOBWNsx6MRVk= 2023-09-29T22:23:41.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J03bPWZgI4AY2VFDOBWNsx6MRVk=', 'Content-Length': '228'} 2023-09-29T22:23:41.757 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd91e832f6020761-0065174e6d-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:41.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:41.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:41.758 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:41.810 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:41.810 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:41.842 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:41.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026221.757224_130.1", 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:23:41.757224Z" 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:41.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:41.857 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:23:41.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:41.857 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:23:41.858 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:41.858 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:23:41.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:41.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:41.858 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:41.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:41.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:41.939 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:41.939+0000 7f7844c0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:41.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:41.940+0000 7f7844c0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:41.952 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:23:41.952 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:23:41.952 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:23:41.952 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:23:41.952 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:41.953 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:41.954 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026221.753757_129.1", 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:23:41.753757Z", 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:41.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:23:41.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:41.958 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:23:41Z" 2023-09-29T22:23:41.959 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:23:41.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026221.753757_129.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:41.753757Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:41Z" } 2023-09-29T22:23:41.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026221.753757_129.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:41.753757Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:23:41.960 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:41.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.753757_129.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:41.960 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026221.757224_130.1 target=1_1696026221.753757_129.1 2023-09-29T22:23:46.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:46.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:46.962 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:47.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:47.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:47.048 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:47.048+0000 7f9c42c44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:47.049 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:47.049+0000 7f9c42c44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:47.062 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:23:47.062 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:23:47.062 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:23:47.062 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:23:47.063 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:47.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:47.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:47.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:47.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:23:47.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:47.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:47.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026221.757224_130.1", 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:23:41.757224Z", 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:47.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:23:47.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:23:47.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:23:47.072 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:23:47.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:23:47.072 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:23:47Z" 2023-09-29T22:23:47.072 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:23:47.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026221.757224_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:41.757224Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:47Z" } 2023-09-29T22:23:47.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026221.757224_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:41.757224Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:23:47.073 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:47.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:47.073 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:23:47.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:23:47.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:47.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:47.074 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:47.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:47.162 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:47.162+0000 7fc6b6b29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:47.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:47.163+0000 7fc6b6b29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:47.181 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:23:47.181 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:23:47.181 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:23:47.181 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:23:47.181 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:23:47.182 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:47.183 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:47.184 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026221.757224_130.1", 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:23:41.757224Z", 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:23:47.186 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:23:47Z" 2023-09-29T22:23:47.187 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:23:47.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026221.757224_130.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:41.757224Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:47Z" } 2023-09-29T22:23:47.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026221.757224_130.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:41.757224Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:23:47.188 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:47.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026221.757224_130.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:23:47.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:23:47.189 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:23:47.189 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:23:47.189 DEBUG:boto:Method: HEAD 2023-09-29T22:23:47.189 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:23:47.189 DEBUG:boto:Data: 2023-09-29T22:23:47.189 DEBUG:boto:Headers: {} 2023-09-29T22:23:47.189 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:47.190 DEBUG:boto:Port: 8000 2023-09-29T22:23:47.190 DEBUG:boto:Params: {} 2023-09-29T22:23:47.190 DEBUG:boto:Token: None 2023-09-29T22:23:47.190 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:47 GMT /dmuhsu-20/ 2023-09-29T22:23:47.190 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TNo4ft4PgPqglaM91TfB6mJ397g= 2023-09-29T22:23:47.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TNo4ft4PgPqglaM91TfB6mJ397g=', 'Content-Length': '0'} 2023-09-29T22:23:47.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb1701a73a82b28c-0065174e73-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:47.193 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:23:47.193 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:23:47.193 DEBUG:boto:Method: PUT 2023-09-29T22:23:47.193 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:23:47.193 DEBUG:boto:Data: 2023-09-29T22:23:47.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:23:47.194 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:23:47.194 DEBUG:boto:Port: 8000 2023-09-29T22:23:47.194 DEBUG:boto:Params: {} 2023-09-29T22:23:47.194 DEBUG:boto:Token: None 2023-09-29T22:23:47.194 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:23:47 GMT /dmuhsu-20/obj 2023-09-29T22:23:47.194 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GKf91xkGaRI/8S2Iv2KMcWtmBlI= 2023-09-29T22:23:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GKf91xkGaRI/8S2Iv2KMcWtmBlI='} 2023-09-29T22:23:47.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-amz-request-id', 'tx00000e9e07f85fc2a30c4-0065174e73-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:47.211 DEBUG:rgw_multi.tests:created new version id=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:23:47.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:23:47.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:47.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:47.211 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:47.266 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:47.266 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:47.302 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:47.302+0000 7f3d77523880 1 waiting for incremental sync to catch up: 2023-09-29T22:23:47.302 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:47.303 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:23:52.304 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:52.303+0000 7f3d77523880 1 waiting for incremental sync to catch up: 2023-09-29T22:23:52.304 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:52.304 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:23:57.305 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:57.305+0000 7f3d77523880 1 bucket sync caught up with source: 2023-09-29T22:23:57.305 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:23:57.305 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:23:57.305 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:23:57.305+0000 7f3d77523880 0 bucket checkpoint complete 2023-09-29T22:23:57.320 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:23:57.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:57.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:57.321 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:57.381 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:57.381 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:23:57.418+0000 7fe333613880 1 waiting for incremental sync to catch up: 2023-09-29T22:23:57.418 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:23:57.418 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:02.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.419+0000 7fe333613880 1 bucket sync caught up with source: 2023-09-29T22:24:02.419 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:02.419 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:02.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.419+0000 7fe333613880 0 bucket checkpoint complete 2023-09-29T22:24:02.434 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:24:02.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.435 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:02.492 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:02.492 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:02.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.528+0000 7f9514baf880 1 bucket sync caught up with source: 2023-09-29T22:24:02.528 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.24.13, , ] 2023-09-29T22:24:02.528 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.24.13, , ] 2023-09-29T22:24:02.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.528+0000 7f9514baf880 0 bucket checkpoint complete 2023-09-29T22:24:02.541 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:24:02.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.542 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:02.594 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:02.595 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:02.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.630+0000 7fa74471e880 1 bucket sync caught up with source: 2023-09-29T22:24:02.630 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.24.13, , ] 2023-09-29T22:24:02.630 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.24.13, , ] 2023-09-29T22:24:02.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.630+0000 7fa74471e880 0 bucket checkpoint complete 2023-09-29T22:24:02.642 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:24:02.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:02.643 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:02.694 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:02.694 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:02.727 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:02.727+0000 7fdaf982e880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:02.727 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:24:02.727 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:07.728 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:07.728+0000 7fdaf982e880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:07.729 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:24:07.729 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:12.730 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:12.730+0000 7fdaf982e880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:12.730 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:24:12.730 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:17.731 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:17.731+0000 7fdaf982e880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:17.731 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-09-29T22:24:17.731 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:22.732 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:22.732+0000 7fdaf982e880 1 bucket sync caught up with source: 2023-09-29T22:24:22.733 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:22.733 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:22.733 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:22.732+0000 7fdaf982e880 0 bucket checkpoint complete 2023-09-29T22:24:22.749 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:24:22.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:22.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:22.749 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:22.808 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:22.808 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:22.845 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:22.845+0000 7f5649b23880 1 bucket sync caught up with source: 2023-09-29T22:24:22.845 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:22.845 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:22.846 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:22.845+0000 7f5649b23880 0 bucket checkpoint complete 2023-09-29T22:24:22.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:24:22.861 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.861 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.861 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.861 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:22.862 DEBUG:boto:Data: 2023-09-29T22:24:22.862 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.862 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.862 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.862 DEBUG:boto:Params: {} 2023-09-29T22:24:22.862 DEBUG:boto:Token: None 2023-09-29T22:24:22.862 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/ 2023-09-29T22:24:22.863 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DD6Ugbpor7UyxzDO5OE9/2EVmko= 2023-09-29T22:24:22.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DD6Ugbpor7UyxzDO5OE9/2EVmko=', 'Content-Length': '0'} 2023-09-29T22:24:22.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d002d97596309df-0065174e96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.865 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.865 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.865 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.866 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:22.866 DEBUG:boto:Data: 2023-09-29T22:24:22.866 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.866 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.866 DEBUG:boto:Params: {} 2023-09-29T22:24:22.866 DEBUG:boto:Token: None 2023-09-29T22:24:22.866 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/ 2023-09-29T22:24:22.866 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DD6Ugbpor7UyxzDO5OE9/2EVmko= 2023-09-29T22:24:22.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DD6Ugbpor7UyxzDO5OE9/2EVmko=', 'Content-Length': '0'} 2023-09-29T22:24:22.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d380b5197fe3cf4e-0065174e96-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:22.869 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.870 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.870 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.870 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.870 DEBUG:boto:Method: GET 2023-09-29T22:24:22.870 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.870 DEBUG:boto:Data: 2023-09-29T22:24:22.870 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.870 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.870 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.870 DEBUG:boto:Params: {} 2023-09-29T22:24:22.871 DEBUG:boto:Token: None 2023-09-29T22:24:22.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/?versions 2023-09-29T22:24:22.871 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54= 2023-09-29T22:24:22.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54=', 'Content-Length': '0'} 2023-09-29T22:24:22.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb58454e52ff0824-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.873 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:22.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:22.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:22.874 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.874 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.875 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.875 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.875 DEBUG:boto:Method: GET 2023-09-29T22:24:22.875 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.875 DEBUG:boto:Data: 2023-09-29T22:24:22.875 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.875 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.875 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.875 DEBUG:boto:Params: {} 2023-09-29T22:24:22.876 DEBUG:boto:Token: None 2023-09-29T22:24:22.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/?versions 2023-09-29T22:24:22.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54= 2023-09-29T22:24:22.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54=', 'Content-Length': '0'} 2023-09-29T22:24:22.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1de30a73d22dc9e-0065174e96-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.879 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:22.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:22.879 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.879 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.879 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.880 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.880 DEBUG:boto:Method: GET 2023-09-29T22:24:22.880 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.880 DEBUG:boto:Data: 2023-09-29T22:24:22.880 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.880 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.880 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.880 DEBUG:boto:Params: {} 2023-09-29T22:24:22.880 DEBUG:boto:Token: None 2023-09-29T22:24:22.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/?versions 2023-09-29T22:24:22.881 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54= 2023-09-29T22:24:22.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54=', 'Content-Length': '0'} 2023-09-29T22:24:22.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049af95f7a6805238-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.883 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.883 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.884 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.884 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.884 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.884 DEBUG:boto:Method: GET 2023-09-29T22:24:22.884 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:22.884 DEBUG:boto:Data: 2023-09-29T22:24:22.884 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.884 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.884 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.884 DEBUG:boto:Params: {} 2023-09-29T22:24:22.885 DEBUG:boto:Token: None 2023-09-29T22:24:22.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/?versions 2023-09-29T22:24:22.885 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54= 2023-09-29T22:24:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DRkEtg6T7bCdF3chHjJrRFinE54=', 'Content-Length': '0'} 2023-09-29T22:24:22.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154fae87ef5862d2-0065174e96-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.930 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:22.931 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.931 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.931 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.931 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.932 DEBUG:boto:Method: GET 2023-09-29T22:24:22.932 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.932 DEBUG:boto:Data: 2023-09-29T22:24:22.932 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.932 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.932 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.932 DEBUG:boto:Params: {} 2023-09-29T22:24:22.932 DEBUG:boto:Token: None 2023-09-29T22:24:22.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000984b5dac96bac11e-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.935 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.935 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.935 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.935 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.935 DEBUG:boto:Method: GET 2023-09-29T22:24:22.935 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.935 DEBUG:boto:Data: 2023-09-29T22:24:22.936 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.936 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.936 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.936 DEBUG:boto:Params: {} 2023-09-29T22:24:22.936 DEBUG:boto:Token: None 2023-09-29T22:24:22.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2f21ca5fea8f4e8-0065174e96-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.938 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.938 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.938 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.938 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.939 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.939 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.939 DEBUG:boto:Data: 2023-09-29T22:24:22.939 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.939 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.939 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.939 DEBUG:boto:Params: {} 2023-09-29T22:24:22.939 DEBUG:boto:Token: None 2023-09-29T22:24:22.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4GepeSBreDoRXcYBnx65YhiXIpU= 2023-09-29T22:24:22.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4GepeSBreDoRXcYBnx65YhiXIpU=', 'Content-Length': '0'} 2023-09-29T22:24:22.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f89da7c08a8fa9e-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.941 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.941 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.941 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.942 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.942 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.942 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.942 DEBUG:boto:Data: 2023-09-29T22:24:22.942 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.942 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.942 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.942 DEBUG:boto:Params: {} 2023-09-29T22:24:22.942 DEBUG:boto:Token: None 2023-09-29T22:24:22.943 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.943 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4GepeSBreDoRXcYBnx65YhiXIpU= 2023-09-29T22:24:22.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4GepeSBreDoRXcYBnx65YhiXIpU=', 'Content-Length': '0'} 2023-09-29T22:24:22.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004443fb27052cdae4-0065174e96-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:22.945 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.945 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.945 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.945 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.945 DEBUG:boto:Method: GET 2023-09-29T22:24:22.946 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.946 DEBUG:boto:Data: 2023-09-29T22:24:22.946 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.946 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.946 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.946 DEBUG:boto:Params: {} 2023-09-29T22:24:22.946 DEBUG:boto:Token: None 2023-09-29T22:24:22.946 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.947 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078554291e9b7b7e0-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.948 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.948 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.949 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.949 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.949 DEBUG:boto:Method: GET 2023-09-29T22:24:22.949 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.949 DEBUG:boto:Data: 2023-09-29T22:24:22.949 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.949 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.949 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.949 DEBUG:boto:Params: {} 2023-09-29T22:24:22.950 DEBUG:boto:Token: None 2023-09-29T22:24:22.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.950 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d00f1505dc358c3-0065174e96-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.952 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.952 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.952 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.952 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.952 DEBUG:boto:Method: GET 2023-09-29T22:24:22.952 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.953 DEBUG:boto:Data: 2023-09-29T22:24:22.953 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.953 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.953 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.953 DEBUG:boto:Params: {} 2023-09-29T22:24:22.953 DEBUG:boto:Token: None 2023-09-29T22:24:22.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/ 2023-09-29T22:24:22.953 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg= 2023-09-29T22:24:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg=', 'Content-Length': '0'} 2023-09-29T22:24:22.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0120c2961742669-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.955 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.956 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.956 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.956 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.956 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.956 DEBUG:boto:Method: GET 2023-09-29T22:24:22.956 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.957 DEBUG:boto:Data: 2023-09-29T22:24:22.957 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.957 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.957 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.957 DEBUG:boto:Params: {} 2023-09-29T22:24:22.957 DEBUG:boto:Token: None 2023-09-29T22:24:22.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/ 2023-09-29T22:24:22.957 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg= 2023-09-29T22:24:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg=', 'Content-Length': '0'} 2023-09-29T22:24:22.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2925dd2647663e-0065174e96-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.962 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:22.962 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.963 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.963 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.963 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:22.963 DEBUG:boto:Data: 2023-09-29T22:24:22.963 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.963 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.963 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.963 DEBUG:boto:Params: {} 2023-09-29T22:24:22.963 DEBUG:boto:Token: None 2023-09-29T22:24:22.964 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj 2023-09-29T22:24:22.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RpCkcgjYpTSYTRsg93Yh8xN6FaI= 2023-09-29T22:24:22.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RpCkcgjYpTSYTRsg93Yh8xN6FaI=', 'Content-Length': '0'} 2023-09-29T22:24:22.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000366670e22561026c-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.966 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.966 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.966 DEBUG:boto:Method: HEAD 2023-09-29T22:24:22.966 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:22.966 DEBUG:boto:Data: 2023-09-29T22:24:22.966 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.966 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.966 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.967 DEBUG:boto:Params: {} 2023-09-29T22:24:22.967 DEBUG:boto:Token: None 2023-09-29T22:24:22.967 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj 2023-09-29T22:24:22.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RpCkcgjYpTSYTRsg93Yh8xN6FaI= 2023-09-29T22:24:22.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RpCkcgjYpTSYTRsg93Yh8xN6FaI=', 'Content-Length': '0'} 2023-09-29T22:24:22.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017623ab60670743e-0065174e96-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:22.970 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.970 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.970 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.970 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.970 DEBUG:boto:Method: GET 2023-09-29T22:24:22.970 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.970 DEBUG:boto:Data: 2023-09-29T22:24:22.970 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.970 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.970 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.971 DEBUG:boto:Params: {} 2023-09-29T22:24:22.971 DEBUG:boto:Token: None 2023-09-29T22:24:22.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.971 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.973 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9c8c855f290b69d-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.973 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:22.973 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:22.973 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.973 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.973 DEBUG:boto:Method: GET 2023-09-29T22:24:22.973 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.974 DEBUG:boto:Data: 2023-09-29T22:24:22.974 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.974 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:22.974 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.974 DEBUG:boto:Params: {} 2023-09-29T22:24:22.974 DEBUG:boto:Token: None 2023-09-29T22:24:22.974 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:22.974 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0= 2023-09-29T22:24:22.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZQYrXv86yhXoyj23TIVQv7P+xn0=', 'Content-Length': '0'} 2023-09-29T22:24:22.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000848fe793751bbea7-0065174e96-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:22.976 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:22.976 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:22.976 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.977 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.977 DEBUG:boto:Method: GET 2023-09-29T22:24:22.977 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:22.977 DEBUG:boto:Data: 2023-09-29T22:24:22.977 DEBUG:boto:Headers: {} 2023-09-29T22:24:22.977 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:22.977 DEBUG:boto:Port: 8000 2023-09-29T22:24:22.977 DEBUG:boto:Params: {} 2023-09-29T22:24:22.977 DEBUG:boto:Token: None 2023-09-29T22:24:22.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:22 GMT /dmuhsu-20/ 2023-09-29T22:24:22.978 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg= 2023-09-29T22:24:22.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:22 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BtDzUGTS2KQKOyGAEErlKYLpHjg=', 'Content-Length': '0'} 2023-09-29T22:24:22.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f34a671d666d163-0065174e96-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.026 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.027 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.027 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.027 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.027 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.027 DEBUG:boto:Method: GET 2023-09-29T22:24:23.027 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.028 DEBUG:boto:Data: 2023-09-29T22:24:23.028 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.028 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.028 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.028 DEBUG:boto:Params: {} 2023-09-29T22:24:23.028 DEBUG:boto:Token: None 2023-09-29T22:24:23.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c6d9d436f3c279-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.031 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.032 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.032 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.032 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.032 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.032 DEBUG:boto:Method: GET 2023-09-29T22:24:23.032 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.032 DEBUG:boto:Data: 2023-09-29T22:24:23.032 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.032 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.032 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.033 DEBUG:boto:Params: {} 2023-09-29T22:24:23.033 DEBUG:boto:Token: None 2023-09-29T22:24:23.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.033 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002078a661a004a149-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.035 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.035 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.035 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.035 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.035 DEBUG:boto:Method: GET 2023-09-29T22:24:23.035 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.035 DEBUG:boto:Data: 2023-09-29T22:24:23.035 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.036 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.036 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.036 DEBUG:boto:Params: {} 2023-09-29T22:24:23.036 DEBUG:boto:Token: None 2023-09-29T22:24:23.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.036 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068826fe8f4fa47cb-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.038 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.038 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.039 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.039 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.039 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.039 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.039 DEBUG:boto:Data: 2023-09-29T22:24:23.039 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.039 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.039 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.039 DEBUG:boto:Params: {} 2023-09-29T22:24:23.039 DEBUG:boto:Token: None 2023-09-29T22:24:23.040 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.040 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069c8dc0cdcabf782-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.042 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.042 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.042 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.042 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.042 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.042 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.042 DEBUG:boto:Data: 2023-09-29T22:24:23.042 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.042 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.042 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.043 DEBUG:boto:Params: {} 2023-09-29T22:24:23.043 DEBUG:boto:Token: None 2023-09-29T22:24:23.043 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1950f439be9bdf2-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.045 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.045 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.046 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.046 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.046 DEBUG:boto:Method: GET 2023-09-29T22:24:23.046 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.046 DEBUG:boto:Data: 2023-09-29T22:24:23.046 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.046 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.046 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.046 DEBUG:boto:Params: {} 2023-09-29T22:24:23.046 DEBUG:boto:Token: None 2023-09-29T22:24:23.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b688d4bac86bf77a-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.049 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.049 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.049 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.049 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.049 DEBUG:boto:Method: GET 2023-09-29T22:24:23.049 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.049 DEBUG:boto:Data: 2023-09-29T22:24:23.049 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.049 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.049 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.050 DEBUG:boto:Params: {} 2023-09-29T22:24:23.050 DEBUG:boto:Token: None 2023-09-29T22:24:23.050 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.050 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a543529938be6bd-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.052 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.052 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.052 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.052 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.053 DEBUG:boto:Method: GET 2023-09-29T22:24:23.053 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.053 DEBUG:boto:Data: 2023-09-29T22:24:23.053 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.053 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.053 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.053 DEBUG:boto:Params: {} 2023-09-29T22:24:23.053 DEBUG:boto:Token: None 2023-09-29T22:24:23.053 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.054 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce7d5597530aa896-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.056 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.056 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.056 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.056 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.056 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.057 DEBUG:boto:Method: GET 2023-09-29T22:24:23.057 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.057 DEBUG:boto:Data: 2023-09-29T22:24:23.057 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.057 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.057 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.057 DEBUG:boto:Params: {} 2023-09-29T22:24:23.057 DEBUG:boto:Token: None 2023-09-29T22:24:23.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052321bbe66afdb79-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.102 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.103 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.103 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.103 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.103 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.103 DEBUG:boto:Data: 2023-09-29T22:24:23.103 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.103 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.103 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.104 DEBUG:boto:Params: {} 2023-09-29T22:24:23.104 DEBUG:boto:Token: None 2023-09-29T22:24:23.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000133aa1e98fcd9f5e-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.106 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.106 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.107 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.107 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.107 DEBUG:boto:Data: 2023-09-29T22:24:23.107 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.107 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.107 DEBUG:boto:Params: {} 2023-09-29T22:24:23.107 DEBUG:boto:Token: None 2023-09-29T22:24:23.107 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.107 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed96033471049083-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.110 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.110 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.110 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.110 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.110 DEBUG:boto:Method: GET 2023-09-29T22:24:23.110 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.110 DEBUG:boto:Data: 2023-09-29T22:24:23.111 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.111 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.111 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.111 DEBUG:boto:Params: {} 2023-09-29T22:24:23.111 DEBUG:boto:Token: None 2023-09-29T22:24:23.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004200c0cc76e16943-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.113 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.113 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.113 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.113 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.114 DEBUG:boto:Method: GET 2023-09-29T22:24:23.114 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.114 DEBUG:boto:Data: 2023-09-29T22:24:23.114 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.114 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.114 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.114 DEBUG:boto:Params: {} 2023-09-29T22:24:23.114 DEBUG:boto:Token: None 2023-09-29T22:24:23.114 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.115 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000481689715ed9de33-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.116 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.116 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.117 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.117 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.117 DEBUG:boto:Method: GET 2023-09-29T22:24:23.117 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.117 DEBUG:boto:Data: 2023-09-29T22:24:23.117 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.117 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.117 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.117 DEBUG:boto:Params: {} 2023-09-29T22:24:23.117 DEBUG:boto:Token: None 2023-09-29T22:24:23.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.118 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d43841f7a2082236-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.120 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.120 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.120 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.120 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.121 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.121 DEBUG:boto:Method: GET 2023-09-29T22:24:23.121 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.121 DEBUG:boto:Data: 2023-09-29T22:24:23.121 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.121 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.121 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.121 DEBUG:boto:Params: {} 2023-09-29T22:24:23.121 DEBUG:boto:Token: None 2023-09-29T22:24:23.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.122 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76424f9c1e16791-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.124 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.125 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:24:23.125 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:24:23.125 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.125 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.125 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.125 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:23.125 DEBUG:boto:Data: 2023-09-29T22:24:23.126 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.126 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.126 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.126 DEBUG:boto:Params: {} 2023-09-29T22:24:23.126 DEBUG:boto:Token: None 2023-09-29T22:24:23.126 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.126 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8= 2023-09-29T22:24:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8=', 'Content-Length': '0'} 2023-09-29T22:24:23.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c87eded660b78ba6-0065174e97-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.129 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.129 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.129 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.129 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:23.130 DEBUG:boto:Data: 2023-09-29T22:24:23.130 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.130 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.130 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.130 DEBUG:boto:Params: {} 2023-09-29T22:24:23.130 DEBUG:boto:Token: None 2023-09-29T22:24:23.130 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.130 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8= 2023-09-29T22:24:23.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8=', 'Content-Length': '0'} 2023-09-29T22:24:23.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082a3cbec94b54968-0065174e97-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.133 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:23.133 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.133 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.133 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.133 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.134 DEBUG:boto:Method: GET 2023-09-29T22:24:23.134 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.134 DEBUG:boto:Data: 2023-09-29T22:24:23.134 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.134 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.134 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.134 DEBUG:boto:Params: {} 2023-09-29T22:24:23.134 DEBUG:boto:Token: None 2023-09-29T22:24:23.134 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1b812d4910bed4-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.137 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.138 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:23.138 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.138 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.138 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.138 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.138 DEBUG:boto:Method: GET 2023-09-29T22:24:23.138 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.139 DEBUG:boto:Data: 2023-09-29T22:24:23.139 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.139 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.139 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.139 DEBUG:boto:Params: {} 2023-09-29T22:24:23.139 DEBUG:boto:Token: None 2023-09-29T22:24:23.139 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d462368fa66afae-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.144 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.144 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.145 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.145 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.145 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.145 DEBUG:boto:Method: GET 2023-09-29T22:24:23.145 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.145 DEBUG:boto:Data: 2023-09-29T22:24:23.145 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.145 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.145 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.145 DEBUG:boto:Params: {} 2023-09-29T22:24:23.146 DEBUG:boto:Token: None 2023-09-29T22:24:23.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.146 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd117ff29b8c2af6-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.148 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.149 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.149 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.149 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.149 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.149 DEBUG:boto:Method: GET 2023-09-29T22:24:23.149 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.149 DEBUG:boto:Data: 2023-09-29T22:24:23.150 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.150 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.150 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.150 DEBUG:boto:Params: {} 2023-09-29T22:24:23.150 DEBUG:boto:Token: None 2023-09-29T22:24:23.150 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.150 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012dd85dd4f895f87-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.153 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.153 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.153 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.154 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.154 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.154 DEBUG:boto:Method: GET 2023-09-29T22:24:23.154 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.154 DEBUG:boto:Data: 2023-09-29T22:24:23.154 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.154 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.154 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.154 DEBUG:boto:Params: {} 2023-09-29T22:24:23.154 DEBUG:boto:Token: None 2023-09-29T22:24:23.155 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa09b25f1d16e810-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.157 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.157 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.157 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.157 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.157 DEBUG:boto:Method: GET 2023-09-29T22:24:23.157 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.157 DEBUG:boto:Data: 2023-09-29T22:24:23.158 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.158 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.158 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.158 DEBUG:boto:Params: {} 2023-09-29T22:24:23.158 DEBUG:boto:Token: None 2023-09-29T22:24:23.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.158 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007484a8471076b948-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.160 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.160 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.160 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.160 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.160 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.161 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.161 DEBUG:boto:Data: 2023-09-29T22:24:23.161 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.161 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.161 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.161 DEBUG:boto:Params: {} 2023-09-29T22:24:23.161 DEBUG:boto:Token: None 2023-09-29T22:24:23.161 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.161 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao= 2023-09-29T22:24:23.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao=', 'Content-Length': '0'} 2023-09-29T22:24:23.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e04fc021a673533-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.163 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.164 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.164 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.164 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.164 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.164 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.164 DEBUG:boto:Data: 2023-09-29T22:24:23.164 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.164 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.164 DEBUG:boto:Params: {} 2023-09-29T22:24:23.164 DEBUG:boto:Token: None 2023-09-29T22:24:23.165 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao= 2023-09-29T22:24:23.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao=', 'Content-Length': '0'} 2023-09-29T22:24:23.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aabad73bf628a7a8-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.167 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.167 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.167 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.167 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.167 DEBUG:boto:Method: GET 2023-09-29T22:24:23.167 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.167 DEBUG:boto:Data: 2023-09-29T22:24:23.167 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.167 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.168 DEBUG:boto:Params: {} 2023-09-29T22:24:23.168 DEBUG:boto:Token: None 2023-09-29T22:24:23.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006be400ebdc62bf21-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.170 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.170 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.170 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.170 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.170 DEBUG:boto:Method: GET 2023-09-29T22:24:23.170 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.170 DEBUG:boto:Data: 2023-09-29T22:24:23.170 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.171 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.171 DEBUG:boto:Params: {} 2023-09-29T22:24:23.171 DEBUG:boto:Token: None 2023-09-29T22:24:23.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.171 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc4ba1a4a197fd9f-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.173 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.173 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.173 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.173 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.173 DEBUG:boto:Method: GET 2023-09-29T22:24:23.173 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.173 DEBUG:boto:Data: 2023-09-29T22:24:23.174 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.174 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.174 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.174 DEBUG:boto:Params: {} 2023-09-29T22:24:23.174 DEBUG:boto:Token: None 2023-09-29T22:24:23.174 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.174 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e3fb690b8d80e20-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.218 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.219 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.219 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.219 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.219 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.219 DEBUG:boto:Method: GET 2023-09-29T22:24:23.219 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.219 DEBUG:boto:Data: 2023-09-29T22:24:23.219 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.220 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.220 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.220 DEBUG:boto:Params: {} 2023-09-29T22:24:23.220 DEBUG:boto:Token: None 2023-09-29T22:24:23.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f65bf2c2c3d1c434-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.223 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.223 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.223 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.223 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.223 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.223 DEBUG:boto:Data: 2023-09-29T22:24:23.224 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.224 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.224 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.224 DEBUG:boto:Params: {} 2023-09-29T22:24:23.224 DEBUG:boto:Token: None 2023-09-29T22:24:23.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a9f808290deaca6-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.226 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.227 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.227 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.227 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.227 DEBUG:boto:Data: 2023-09-29T22:24:23.227 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.227 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.227 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.227 DEBUG:boto:Params: {} 2023-09-29T22:24:23.227 DEBUG:boto:Token: None 2023-09-29T22:24:23.228 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000983812a7b1c3c43e-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.230 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.230 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.230 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.231 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.231 DEBUG:boto:Method: GET 2023-09-29T22:24:23.231 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.231 DEBUG:boto:Data: 2023-09-29T22:24:23.231 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.231 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.231 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.231 DEBUG:boto:Params: {} 2023-09-29T22:24:23.231 DEBUG:boto:Token: None 2023-09-29T22:24:23.231 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000483fc2c40319e95c-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.233 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.234 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.234 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.234 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.234 DEBUG:boto:Method: GET 2023-09-29T22:24:23.234 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.234 DEBUG:boto:Data: 2023-09-29T22:24:23.234 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.234 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.234 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.234 DEBUG:boto:Params: {} 2023-09-29T22:24:23.235 DEBUG:boto:Token: None 2023-09-29T22:24:23.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.235 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4092514b8a24386-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.237 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.237 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.237 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.237 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.237 DEBUG:boto:Method: GET 2023-09-29T22:24:23.238 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.238 DEBUG:boto:Data: 2023-09-29T22:24:23.238 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.238 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.238 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.238 DEBUG:boto:Params: {} 2023-09-29T22:24:23.238 DEBUG:boto:Token: None 2023-09-29T22:24:23.238 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.238 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c74e18f9935e418b-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.241 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.241 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.241 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.241 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.241 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.241 DEBUG:boto:Method: GET 2023-09-29T22:24:23.242 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.242 DEBUG:boto:Data: 2023-09-29T22:24:23.242 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.242 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.242 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.242 DEBUG:boto:Params: {} 2023-09-29T22:24:23.242 DEBUG:boto:Token: None 2023-09-29T22:24:23.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.242 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001914cb1f63f430fb-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.245 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.245 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.245 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.245 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.245 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.246 DEBUG:boto:Method: GET 2023-09-29T22:24:23.246 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.246 DEBUG:boto:Data: 2023-09-29T22:24:23.246 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.246 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.246 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.246 DEBUG:boto:Params: {} 2023-09-29T22:24:23.246 DEBUG:boto:Token: None 2023-09-29T22:24:23.246 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.247 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073d6d0bf68ec14eb-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.248 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.248 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.249 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.249 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.249 DEBUG:boto:Method: GET 2023-09-29T22:24:23.249 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.249 DEBUG:boto:Data: 2023-09-29T22:24:23.249 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.249 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.249 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.249 DEBUG:boto:Params: {} 2023-09-29T22:24:23.249 DEBUG:boto:Token: None 2023-09-29T22:24:23.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.250 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b81720310c3d1e9-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.251 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.252 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.252 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.252 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.252 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.252 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.252 DEBUG:boto:Data: 2023-09-29T22:24:23.252 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.252 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.252 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.252 DEBUG:boto:Params: {} 2023-09-29T22:24:23.252 DEBUG:boto:Token: None 2023-09-29T22:24:23.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.253 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4eed88925dc0d17-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.255 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.255 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.255 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.255 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.255 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.255 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.256 DEBUG:boto:Data: 2023-09-29T22:24:23.256 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.256 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.256 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.256 DEBUG:boto:Params: {} 2023-09-29T22:24:23.256 DEBUG:boto:Token: None 2023-09-29T22:24:23.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebdfe1da3f4d9b18-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.259 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.259 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.259 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.260 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.260 DEBUG:boto:Method: GET 2023-09-29T22:24:23.260 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.260 DEBUG:boto:Data: 2023-09-29T22:24:23.261 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.261 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.261 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.261 DEBUG:boto:Params: {} 2023-09-29T22:24:23.261 DEBUG:boto:Token: None 2023-09-29T22:24:23.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.262 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c46c830d4e70e2b9-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.265 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.265 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.265 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.265 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.265 DEBUG:boto:Method: GET 2023-09-29T22:24:23.266 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.266 DEBUG:boto:Data: 2023-09-29T22:24:23.266 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.266 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.266 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.266 DEBUG:boto:Params: {} 2023-09-29T22:24:23.266 DEBUG:boto:Token: None 2023-09-29T22:24:23.267 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002ddb133c5c2d7f9-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.269 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.269 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.269 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.269 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.269 DEBUG:boto:Method: GET 2023-09-29T22:24:23.269 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.270 DEBUG:boto:Data: 2023-09-29T22:24:23.270 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.270 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.270 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.270 DEBUG:boto:Params: {} 2023-09-29T22:24:23.270 DEBUG:boto:Token: None 2023-09-29T22:24:23.270 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000503c757e219ac95-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.273 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.273 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.274 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.274 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.274 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.274 DEBUG:boto:Method: GET 2023-09-29T22:24:23.274 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.274 DEBUG:boto:Data: 2023-09-29T22:24:23.274 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.275 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.275 DEBUG:boto:Params: {} 2023-09-29T22:24:23.275 DEBUG:boto:Token: None 2023-09-29T22:24:23.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c214ab52aec479-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.278 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.278 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.278 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.279 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.279 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.279 DEBUG:boto:Data: 2023-09-29T22:24:23.279 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.279 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.279 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.279 DEBUG:boto:Params: {} 2023-09-29T22:24:23.279 DEBUG:boto:Token: None 2023-09-29T22:24:23.280 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.280 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc510259e3a18a5e-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.282 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.282 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.282 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.282 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.283 DEBUG:boto:Data: 2023-09-29T22:24:23.283 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.283 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.283 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.283 DEBUG:boto:Params: {} 2023-09-29T22:24:23.283 DEBUG:boto:Token: None 2023-09-29T22:24:23.283 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.283 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a28915fb21e302c-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.286 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.286 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.286 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.286 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.286 DEBUG:boto:Method: GET 2023-09-29T22:24:23.286 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.287 DEBUG:boto:Data: 2023-09-29T22:24:23.287 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.287 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.287 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.287 DEBUG:boto:Params: {} 2023-09-29T22:24:23.287 DEBUG:boto:Token: None 2023-09-29T22:24:23.287 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.287 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093eb604c34535ee2-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.289 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.289 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.290 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.290 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.290 DEBUG:boto:Method: GET 2023-09-29T22:24:23.290 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.290 DEBUG:boto:Data: 2023-09-29T22:24:23.290 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.290 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.290 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.291 DEBUG:boto:Params: {} 2023-09-29T22:24:23.291 DEBUG:boto:Token: None 2023-09-29T22:24:23.291 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.291 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009027a887f6b6aeb8-0065174e97-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.293 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.293 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.293 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.294 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.294 DEBUG:boto:Method: GET 2023-09-29T22:24:23.294 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.294 DEBUG:boto:Data: 2023-09-29T22:24:23.294 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.294 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.294 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.294 DEBUG:boto:Params: {} 2023-09-29T22:24:23.295 DEBUG:boto:Token: None 2023-09-29T22:24:23.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.295 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007def3575cf829f87-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.297 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.297 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.298 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.298 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.298 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.298 DEBUG:boto:Method: GET 2023-09-29T22:24:23.298 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.298 DEBUG:boto:Data: 2023-09-29T22:24:23.298 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.298 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.299 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.299 DEBUG:boto:Params: {} 2023-09-29T22:24:23.299 DEBUG:boto:Token: None 2023-09-29T22:24:23.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8a6bc7952adebd-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.302 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:24:23.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:24:23.303 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.303 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.303 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.303 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:23.304 DEBUG:boto:Data: 2023-09-29T22:24:23.304 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.304 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.304 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.304 DEBUG:boto:Params: {} 2023-09-29T22:24:23.305 DEBUG:boto:Token: None 2023-09-29T22:24:23.305 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.305 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8= 2023-09-29T22:24:23.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8=', 'Content-Length': '0'} 2023-09-29T22:24:23.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8aaad8a182788ae-0065174e97-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.308 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.308 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.308 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.308 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:23.308 DEBUG:boto:Data: 2023-09-29T22:24:23.308 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.309 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.309 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.309 DEBUG:boto:Params: {} 2023-09-29T22:24:23.309 DEBUG:boto:Token: None 2023-09-29T22:24:23.309 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.310 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8= 2023-09-29T22:24:23.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8=', 'Content-Length': '0'} 2023-09-29T22:24:23.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d311e457ab0830c8-0065174e97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.312 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:23.312 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.312 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.313 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.313 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.313 DEBUG:boto:Method: GET 2023-09-29T22:24:23.313 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.313 DEBUG:boto:Data: 2023-09-29T22:24:23.313 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.313 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.313 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.313 DEBUG:boto:Params: {} 2023-09-29T22:24:23.313 DEBUG:boto:Token: None 2023-09-29T22:24:23.314 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.314 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3ebd81531c23c75-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.358 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:23.359 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.359 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.360 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.360 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.360 DEBUG:boto:Method: GET 2023-09-29T22:24:23.360 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.360 DEBUG:boto:Data: 2023-09-29T22:24:23.360 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.360 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.360 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.360 DEBUG:boto:Params: {} 2023-09-29T22:24:23.360 DEBUG:boto:Token: None 2023-09-29T22:24:23.361 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.361 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e9e3ee6740ac8e-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.363 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:23.364 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.364 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.364 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.364 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.364 DEBUG:boto:Method: GET 2023-09-29T22:24:23.364 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.364 DEBUG:boto:Data: 2023-09-29T22:24:23.365 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.365 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.365 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.365 DEBUG:boto:Params: {} 2023-09-29T22:24:23.365 DEBUG:boto:Token: None 2023-09-29T22:24:23.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.365 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000254184edcbf69ea9-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.367 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.368 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.368 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.368 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.368 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.368 DEBUG:boto:Method: GET 2023-09-29T22:24:23.368 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:23.368 DEBUG:boto:Data: 2023-09-29T22:24:23.369 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.369 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.369 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.369 DEBUG:boto:Params: {} 2023-09-29T22:24:23.369 DEBUG:boto:Token: None 2023-09-29T22:24:23.369 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versions 2023-09-29T22:24:23.369 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA= 2023-09-29T22:24:23.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sG4caQgcnL5GDDpw9nbIzZPzvoA=', 'Content-Length': '0'} 2023-09-29T22:24:23.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e716ae6367ef622-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.418 DEBUG:boto:b'dmuhsu-20999falseobjNv4xektc4xejzvQTQwXkSjB93oMIKJStrue2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:23:00.704Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.419 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.419 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.419 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.419 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.420 DEBUG:boto:Method: GET 2023-09-29T22:24:23.420 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.420 DEBUG:boto:Data: 2023-09-29T22:24:23.420 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.420 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.420 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.420 DEBUG:boto:Params: {} 2023-09-29T22:24:23.420 DEBUG:boto:Token: None 2023-09-29T22:24:23.421 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d347caf6cfb917d-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.423 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.423 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.423 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.423 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.423 DEBUG:boto:Method: GET 2023-09-29T22:24:23.423 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.423 DEBUG:boto:Data: 2023-09-29T22:24:23.423 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.423 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.424 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.424 DEBUG:boto:Params: {} 2023-09-29T22:24:23.424 DEBUG:boto:Token: None 2023-09-29T22:24:23.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bfa613bf6229029-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.426 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.426 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.426 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.426 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.426 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.427 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.427 DEBUG:boto:Data: 2023-09-29T22:24:23.427 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.427 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.427 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.427 DEBUG:boto:Params: {} 2023-09-29T22:24:23.427 DEBUG:boto:Token: None 2023-09-29T22:24:23.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.428 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao= 2023-09-29T22:24:23.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao=', 'Content-Length': '0'} 2023-09-29T22:24:23.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ea78be0b9040a23-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.430 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.430 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.430 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.430 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.430 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.430 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.430 DEBUG:boto:Data: 2023-09-29T22:24:23.430 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.430 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.431 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.431 DEBUG:boto:Params: {} 2023-09-29T22:24:23.431 DEBUG:boto:Token: None 2023-09-29T22:24:23.431 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.431 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao= 2023-09-29T22:24:23.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rW/wKPZ4P4CqDUp5TJqgQCXkhao=', 'Content-Length': '0'} 2023-09-29T22:24:23.433 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000151005e3c9a5b729-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.433 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.433 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.433 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.433 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.434 DEBUG:boto:Method: GET 2023-09-29T22:24:23.434 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.434 DEBUG:boto:Data: 2023-09-29T22:24:23.434 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.434 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.434 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.434 DEBUG:boto:Params: {} 2023-09-29T22:24:23.434 DEBUG:boto:Token: None 2023-09-29T22:24:23.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.435 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032b08383fa534772-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.436 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.436 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.437 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.437 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.437 DEBUG:boto:Method: GET 2023-09-29T22:24:23.437 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.437 DEBUG:boto:Data: 2023-09-29T22:24:23.437 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.437 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.437 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.437 DEBUG:boto:Params: {} 2023-09-29T22:24:23.438 DEBUG:boto:Token: None 2023-09-29T22:24:23.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.438 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a21dff24364e496-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.440 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.440 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.440 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.440 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.440 DEBUG:boto:Method: GET 2023-09-29T22:24:23.441 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.441 DEBUG:boto:Data: 2023-09-29T22:24:23.441 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.441 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.441 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.441 DEBUG:boto:Params: {} 2023-09-29T22:24:23.441 DEBUG:boto:Token: None 2023-09-29T22:24:23.441 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.441 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b0828959d59142-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.444 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.444 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.444 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.444 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.445 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.445 DEBUG:boto:Method: GET 2023-09-29T22:24:23.445 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.445 DEBUG:boto:Data: 2023-09-29T22:24:23.445 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.445 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.445 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.445 DEBUG:boto:Params: {} 2023-09-29T22:24:23.445 DEBUG:boto:Token: None 2023-09-29T22:24:23.446 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.446 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f87e8784942e5d-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.448 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.449 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.449 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.449 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.449 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.449 DEBUG:boto:Data: 2023-09-29T22:24:23.449 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.449 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.449 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.450 DEBUG:boto:Params: {} 2023-09-29T22:24:23.450 DEBUG:boto:Token: None 2023-09-29T22:24:23.450 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.450 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044b0323504a0ef5f-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.452 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.452 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.453 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.453 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.453 DEBUG:boto:Data: 2023-09-29T22:24:23.453 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.453 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.453 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.453 DEBUG:boto:Params: {} 2023-09-29T22:24:23.453 DEBUG:boto:Token: None 2023-09-29T22:24:23.453 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.454 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000327ab7a36374426-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.456 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.456 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.456 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.457 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.457 DEBUG:boto:Method: GET 2023-09-29T22:24:23.457 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.457 DEBUG:boto:Data: 2023-09-29T22:24:23.457 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.457 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.457 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.457 DEBUG:boto:Params: {} 2023-09-29T22:24:23.457 DEBUG:boto:Token: None 2023-09-29T22:24:23.458 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.458 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000586b7ac78ee6f575-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.459 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.459 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.460 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.460 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.460 DEBUG:boto:Method: GET 2023-09-29T22:24:23.460 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.460 DEBUG:boto:Data: 2023-09-29T22:24:23.460 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.460 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.460 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.460 DEBUG:boto:Params: {} 2023-09-29T22:24:23.461 DEBUG:boto:Token: None 2023-09-29T22:24:23.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000460e08b4ee8471c6-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.463 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.463 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.463 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.463 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.463 DEBUG:boto:Method: GET 2023-09-29T22:24:23.463 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.464 DEBUG:boto:Data: 2023-09-29T22:24:23.464 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.464 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.464 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.464 DEBUG:boto:Params: {} 2023-09-29T22:24:23.464 DEBUG:boto:Token: None 2023-09-29T22:24:23.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.464 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d93f548d349648bb-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.514 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.515 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.515 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.515 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.515 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.515 DEBUG:boto:Method: GET 2023-09-29T22:24:23.516 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.516 DEBUG:boto:Data: 2023-09-29T22:24:23.516 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.516 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.516 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.516 DEBUG:boto:Params: {} 2023-09-29T22:24:23.516 DEBUG:boto:Token: None 2023-09-29T22:24:23.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.517 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008acf80b04ed79ccc-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.519 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.520 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.520 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.520 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.520 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.520 DEBUG:boto:Method: GET 2023-09-29T22:24:23.520 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.520 DEBUG:boto:Data: 2023-09-29T22:24:23.521 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.521 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.521 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.521 DEBUG:boto:Params: {} 2023-09-29T22:24:23.521 DEBUG:boto:Token: None 2023-09-29T22:24:23.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.521 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e002364be285136-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.524 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.524 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.524 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.524 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.524 DEBUG:boto:Method: GET 2023-09-29T22:24:23.524 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.524 DEBUG:boto:Data: 2023-09-29T22:24:23.524 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.524 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.525 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.525 DEBUG:boto:Params: {} 2023-09-29T22:24:23.525 DEBUG:boto:Token: None 2023-09-29T22:24:23.525 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.525 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007561614dc2f2cb31-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.527 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.527 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.527 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.528 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.528 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.528 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.528 DEBUG:boto:Data: 2023-09-29T22:24:23.528 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.528 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.528 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.528 DEBUG:boto:Params: {} 2023-09-29T22:24:23.528 DEBUG:boto:Token: None 2023-09-29T22:24:23.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.529 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000393eca9b2dd4e177-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.531 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.531 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.531 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.531 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.531 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.531 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.531 DEBUG:boto:Data: 2023-09-29T22:24:23.531 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.531 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.531 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.532 DEBUG:boto:Params: {} 2023-09-29T22:24:23.532 DEBUG:boto:Token: None 2023-09-29T22:24:23.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.532 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw= 2023-09-29T22:24:23.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eKKhSSMHiwprvU9l4wsOIYSR3iw=', 'Content-Length': '0'} 2023-09-29T22:24:23.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092f9ac81bf74c5e1-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.534 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.534 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.534 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.534 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.535 DEBUG:boto:Method: GET 2023-09-29T22:24:23.535 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.535 DEBUG:boto:Data: 2023-09-29T22:24:23.535 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.535 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.535 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.535 DEBUG:boto:Params: {} 2023-09-29T22:24:23.535 DEBUG:boto:Token: None 2023-09-29T22:24:23.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.535 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acaccbdcd647c331-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.537 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.537 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.538 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.538 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.538 DEBUG:boto:Method: GET 2023-09-29T22:24:23.538 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.538 DEBUG:boto:Data: 2023-09-29T22:24:23.538 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.538 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.538 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.538 DEBUG:boto:Params: {} 2023-09-29T22:24:23.538 DEBUG:boto:Token: None 2023-09-29T22:24:23.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:23.539 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs= 2023-09-29T22:24:23.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S3TooGZUS00SELaTvB+aBkBNuSs=', 'Content-Length': '0'} 2023-09-29T22:24:23.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db9be416c0cb2566-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.540 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.541 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.541 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.541 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.541 DEBUG:boto:Method: GET 2023-09-29T22:24:23.541 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.541 DEBUG:boto:Data: 2023-09-29T22:24:23.541 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.541 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.541 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.541 DEBUG:boto:Params: {} 2023-09-29T22:24:23.541 DEBUG:boto:Token: None 2023-09-29T22:24:23.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.542 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02f8f99243d7eb9-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.586 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.587 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.587 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.587 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.587 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.587 DEBUG:boto:Method: GET 2023-09-29T22:24:23.587 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.587 DEBUG:boto:Data: 2023-09-29T22:24:23.587 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.588 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.588 DEBUG:boto:Params: {} 2023-09-29T22:24:23.588 DEBUG:boto:Token: None 2023-09-29T22:24:23.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0adf1cd35072262-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.590 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.591 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.591 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.591 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.591 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.591 DEBUG:boto:Data: 2023-09-29T22:24:23.591 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.591 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.592 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.592 DEBUG:boto:Params: {} 2023-09-29T22:24:23.592 DEBUG:boto:Token: None 2023-09-29T22:24:23.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.592 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a430b18860ee9eea-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.594 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.594 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.595 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.595 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.595 DEBUG:boto:Data: 2023-09-29T22:24:23.595 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.595 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.595 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.595 DEBUG:boto:Params: {} 2023-09-29T22:24:23.595 DEBUG:boto:Token: None 2023-09-29T22:24:23.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.596 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8= 2023-09-29T22:24:23.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ycc1spY2HflCjDPlhCGwBcQJPe8=', 'Content-Length': '0'} 2023-09-29T22:24:23.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007687cd40414b17f8-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:23.598 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.598 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.598 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.598 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.598 DEBUG:boto:Method: GET 2023-09-29T22:24:23.599 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.599 DEBUG:boto:Data: 2023-09-29T22:24:23.599 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.599 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.599 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.599 DEBUG:boto:Params: {} 2023-09-29T22:24:23.599 DEBUG:boto:Token: None 2023-09-29T22:24:23.599 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.599 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa67ad2f4f99d890-0065174e97-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.601 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.601 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.601 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.602 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.602 DEBUG:boto:Method: GET 2023-09-29T22:24:23.602 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.602 DEBUG:boto:Data: 2023-09-29T22:24:23.602 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.602 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.602 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.602 DEBUG:boto:Params: {} 2023-09-29T22:24:23.602 DEBUG:boto:Token: None 2023-09-29T22:24:23.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:23.603 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ= 2023-09-29T22:24:23.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yJVdPjDZ6y8X4Ly8F65OFwWNvJQ=', 'Content-Length': '0'} 2023-09-29T22:24:23.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cb173efb99fc2de-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.605 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.605 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.605 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.605 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.605 DEBUG:boto:Method: GET 2023-09-29T22:24:23.605 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.605 DEBUG:boto:Data: 2023-09-29T22:24:23.605 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.605 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:23.605 DEBUG:boto:Port: 8001 2023-09-29T22:24:23.606 DEBUG:boto:Params: {} 2023-09-29T22:24:23.606 DEBUG:boto:Token: None 2023-09-29T22:24:23.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.606 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0ea7e8102874689-0065174e97-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.608 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.609 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.609 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.609 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.609 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.609 DEBUG:boto:Method: GET 2023-09-29T22:24:23.609 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:23.609 DEBUG:boto:Data: 2023-09-29T22:24:23.609 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.610 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:23.610 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.610 DEBUG:boto:Params: {} 2023-09-29T22:24:23.610 DEBUG:boto:Token: None 2023-09-29T22:24:23.610 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.610 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o= 2023-09-29T22:24:23.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:97bh/3eJa0OVfJMK2aVsxz3Vw9o=', 'Content-Length': '0'} 2023-09-29T22:24:23.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041db580547734a97-0065174e97-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.613 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:23.613 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:24:23.613 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.613 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.613 DEBUG:boto:path=/dmuhsu-20/?versioning 2023-09-29T22:24:23.613 DEBUG:boto:auth_path=/dmuhsu-20/?versioning 2023-09-29T22:24:23.614 DEBUG:boto:Method: PUT 2023-09-29T22:24:23.614 DEBUG:boto:Path: /dmuhsu-20/?versioning 2023-09-29T22:24:23.614 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:24:23.614 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.614 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.614 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.614 DEBUG:boto:Params: {} 2023-09-29T22:24:23.614 DEBUG:boto:Token: None 2023-09-29T22:24:23.615 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/?versioning 2023-09-29T22:24:23.615 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9/81u7HyEC3T3naXzxJiid1KcAc= 2023-09-29T22:24:23.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9/81u7HyEC3T3naXzxJiid1KcAc=', 'Content-Length': '230'} 2023-09-29T22:24:23.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000805a2e6abfd4397b-0065174e97-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.620 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:24:23.665 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:23.666 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:23.695 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:24:23.707 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:24:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:24:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026263.620480_132.1", 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:24:23.620480Z" 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:24:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:24:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.709 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:24:23.710 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026263.620480_132.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:24:23.710 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:24:23.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.710 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:24:23.764 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:23.764 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:23.799 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:23.799+0000 7fecb2030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:23.800 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:23.800+0000 7fecb2030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:24:23.812 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:24:23.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:24:23.814 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:24:23.815 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026263.620480_132.1", 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:24:23.620480Z", 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:24:23.816 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:24:23.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:24:23Z" 2023-09-29T22:24:23.818 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:24:23.819 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026263.620480_132.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:24:23.620480Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:24:23Z" } 2023-09-29T22:24:23.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026263.620480_132.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:24:23.620480Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:24:23.819 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026263.620480_132.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:24:23.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026263.620480_132.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:24:23.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:24:23.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:24:23.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:23.820 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:24:23.865 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:23.866 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:23.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:23.894+0000 7fdf9b00a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:23.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:23.895+0000 7fdf9b00a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:24:23.905 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:24:23.906 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:24:23.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:24:23.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026263.620480_132.1", 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:24:23.620480Z", 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:24:23.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:24:23Z" 2023-09-29T22:24:23.911 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:24:23.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026263.620480_132.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:24:23.620480Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:24:23Z" } 2023-09-29T22:24:23.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026263.620480_132.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:24:23.620480Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:24:23.912 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026263.620480_132.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:24:23.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026263.620480_132.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:24:23.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:24:23.913 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:23.913 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:23.913 DEBUG:boto:Method: HEAD 2023-09-29T22:24:23.913 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:23.913 DEBUG:boto:Data: 2023-09-29T22:24:23.913 DEBUG:boto:Headers: {} 2023-09-29T22:24:23.913 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.913 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.914 DEBUG:boto:Params: {} 2023-09-29T22:24:23.914 DEBUG:boto:Token: None 2023-09-29T22:24:23.914 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/ 2023-09-29T22:24:23.914 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8= 2023-09-29T22:24:23.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ma1IKs1Cx1nCu/lIzmEZLMFAxM8=', 'Content-Length': '0'} 2023-09-29T22:24:23.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccf00e5514bd0645-0065174e97-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.917 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:23.917 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:23.917 DEBUG:boto:Method: PUT 2023-09-29T22:24:23.917 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:23.917 DEBUG:boto:Data: 2023-09-29T22:24:23.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:24:23.918 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:23.918 DEBUG:boto:Port: 8000 2023-09-29T22:24:23.918 DEBUG:boto:Params: {} 2023-09-29T22:24:23.918 DEBUG:boto:Token: None 2023-09-29T22:24:23.918 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:24:23 GMT /dmuhsu-20/obj 2023-09-29T22:24:23.918 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NWRh5HE5t+H9cRclY/rv2mlmGDQ= 2023-09-29T22:24:23.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:24:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NWRh5HE5t+H9cRclY/rv2mlmGDQ='} 2023-09-29T22:24:23.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af8419a8d553a713-0065174e97-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:23.932 DEBUG:rgw_multi.tests:created null version id=None 2023-09-29T22:24:23.932 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:24:23.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:23.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:23.933 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:23.989 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:23.989 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:24.023 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:24.023+0000 7ff14c127880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:24.023 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:24.023 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:29.025 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:29.025+0000 7ff14c127880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:29.025 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:29.025 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:34.026 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:34.026+0000 7ff14c127880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:34.026 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:34.026 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:39.027 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:39.027+0000 7ff14c127880 1 bucket sync caught up with source: 2023-09-29T22:24:39.028 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:39.028 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:39.028 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:39.027+0000 7ff14c127880 0 bucket checkpoint complete 2023-09-29T22:24:39.043 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:24:39.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:39.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:39.044 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:39.103 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:39.103 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:39.138 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:39.137+0000 7fc71754b880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:39.138 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:24:39.138 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:44.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.139+0000 7fc71754b880 1 bucket sync caught up with source: 2023-09-29T22:24:44.141 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:44.141 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:24:44.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.139+0000 7fc71754b880 0 bucket checkpoint complete 2023-09-29T22:24:44.156 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:24:44.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.157 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:44.213 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:44.214 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:44.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.250+0000 7ff490330880 1 bucket sync caught up with source: 2023-09-29T22:24:44.251 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000006.31.18, , ] 2023-09-29T22:24:44.252 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000006.31.18, , ] 2023-09-29T22:24:44.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.250+0000 7ff490330880 0 bucket checkpoint complete 2023-09-29T22:24:44.264 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-20 2023-09-29T22:24:44.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.265 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:44.316 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:44.350 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.350+0000 7ff8de934880 1 bucket sync caught up with source: 2023-09-29T22:24:44.351 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000006.31.18, , ] 2023-09-29T22:24:44.351 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000006.31.18, , ] 2023-09-29T22:24:44.351 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.350+0000 7ff8de934880 0 bucket checkpoint complete 2023-09-29T22:24:44.362 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-20 2023-09-29T22:24:44.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:44.363 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:44.414 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:44.415 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:44.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:44.447+0000 7f9bb6d27880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:44.448 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:44.448 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:49.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:49.448+0000 7f9bb6d27880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:49.450 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:49.450 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:54.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:54.451+0000 7f9bb6d27880 1 waiting for incremental sync to catch up: 2023-09-29T22:24:54.451 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.24.10, , ] 2023-09-29T22:24:54.451 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:59.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:59.452+0000 7f9bb6d27880 1 bucket sync caught up with source: 2023-09-29T22:24:59.452 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:59.452 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:59.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:24:59.452+0000 7f9bb6d27880 0 bucket checkpoint complete 2023-09-29T22:24:59.468 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-20 2023-09-29T22:24:59.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:59.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:59.469 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:59.564 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:59.564+0000 7f6455a13880 1 bucket sync caught up with source: 2023-09-29T22:24:59.565 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:59.565 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000008.32.15, , ] 2023-09-29T22:24:59.565 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:24:59.564+0000 7f6455a13880 0 bucket checkpoint complete 2023-09-29T22:24:59.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:24:59.578 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.578 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.578 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.578 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:59.578 DEBUG:boto:Data: 2023-09-29T22:24:59.578 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.578 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.579 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.579 DEBUG:boto:Params: {} 2023-09-29T22:24:59.579 DEBUG:boto:Token: None 2023-09-29T22:24:59.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c= 2023-09-29T22:24:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c=', 'Content-Length': '0'} 2023-09-29T22:24:59.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014deeea49c8f801b-0065174ebb-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.582 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.582 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.582 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.582 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:59.582 DEBUG:boto:Data: 2023-09-29T22:24:59.582 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.583 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.583 DEBUG:boto:Params: {} 2023-09-29T22:24:59.583 DEBUG:boto:Token: None 2023-09-29T22:24:59.583 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.583 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c= 2023-09-29T22:24:59.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c=', 'Content-Length': '0'} 2023-09-29T22:24:59.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c53499a06bd4e4aa-0065174ebb-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.586 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:59.587 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.587 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.587 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.587 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.587 DEBUG:boto:Method: GET 2023-09-29T22:24:59.587 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.587 DEBUG:boto:Data: 2023-09-29T22:24:59.587 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.587 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.587 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.588 DEBUG:boto:Params: {} 2023-09-29T22:24:59.588 DEBUG:boto:Token: None 2023-09-29T22:24:59.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.588 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3ae9a9bc9994a97-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.590 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:59.591 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.591 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.591 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.592 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.592 DEBUG:boto:Method: GET 2023-09-29T22:24:59.592 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.592 DEBUG:boto:Data: 2023-09-29T22:24:59.592 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.592 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.592 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.592 DEBUG:boto:Params: {} 2023-09-29T22:24:59.592 DEBUG:boto:Token: None 2023-09-29T22:24:59.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.593 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f168509729685a67-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.596 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.597 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.597 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.597 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.597 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.597 DEBUG:boto:Method: GET 2023-09-29T22:24:59.597 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.597 DEBUG:boto:Data: 2023-09-29T22:24:59.597 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.598 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.598 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.598 DEBUG:boto:Params: {} 2023-09-29T22:24:59.598 DEBUG:boto:Token: None 2023-09-29T22:24:59.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.598 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b520f2b737bcd686-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.600 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.601 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.601 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.601 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.601 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.601 DEBUG:boto:Method: GET 2023-09-29T22:24:59.601 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.602 DEBUG:boto:Data: 2023-09-29T22:24:59.602 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.602 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.602 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.602 DEBUG:boto:Params: {} 2023-09-29T22:24:59.602 DEBUG:boto:Token: None 2023-09-29T22:24:59.602 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010dd3e2816c2c282-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.646 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.647 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.647 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.647 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.647 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.647 DEBUG:boto:Method: GET 2023-09-29T22:24:59.647 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.647 DEBUG:boto:Data: 2023-09-29T22:24:59.648 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.648 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.648 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.648 DEBUG:boto:Params: {} 2023-09-29T22:24:59.648 DEBUG:boto:Token: None 2023-09-29T22:24:59.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.648 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e358fa893ff041ef-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.650 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.650 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.651 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.651 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.651 DEBUG:boto:Method: GET 2023-09-29T22:24:59.651 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.651 DEBUG:boto:Data: 2023-09-29T22:24:59.651 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.651 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.651 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.651 DEBUG:boto:Params: {} 2023-09-29T22:24:59.651 DEBUG:boto:Token: None 2023-09-29T22:24:59.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011b752d86e630fe7-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.654 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.654 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.654 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.654 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.654 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.654 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.654 DEBUG:boto:Data: 2023-09-29T22:24:59.655 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.655 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.655 DEBUG:boto:Params: {} 2023-09-29T22:24:59.655 DEBUG:boto:Token: None 2023-09-29T22:24:59.655 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.655 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw= 2023-09-29T22:24:59.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw=', 'Content-Length': '0'} 2023-09-29T22:24:59.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060b39d87dd898b19-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.657 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.657 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.657 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.657 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.657 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.657 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.657 DEBUG:boto:Data: 2023-09-29T22:24:59.657 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.658 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.658 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.658 DEBUG:boto:Params: {} 2023-09-29T22:24:59.658 DEBUG:boto:Token: None 2023-09-29T22:24:59.658 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.658 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw= 2023-09-29T22:24:59.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw=', 'Content-Length': '0'} 2023-09-29T22:24:59.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000024a9ea358bf881d-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.661 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.661 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.661 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.661 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.661 DEBUG:boto:Method: GET 2023-09-29T22:24:59.661 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.661 DEBUG:boto:Data: 2023-09-29T22:24:59.661 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.661 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.662 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.662 DEBUG:boto:Params: {} 2023-09-29T22:24:59.662 DEBUG:boto:Token: None 2023-09-29T22:24:59.662 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.662 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de8d92dd4717414c-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.664 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.664 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.664 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.664 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.664 DEBUG:boto:Method: GET 2023-09-29T22:24:59.664 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.665 DEBUG:boto:Data: 2023-09-29T22:24:59.665 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.665 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.665 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.665 DEBUG:boto:Params: {} 2023-09-29T22:24:59.665 DEBUG:boto:Token: None 2023-09-29T22:24:59.665 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.665 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c3d698596c177f2-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.667 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.667 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.667 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.668 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.668 DEBUG:boto:Method: GET 2023-09-29T22:24:59.668 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.668 DEBUG:boto:Data: 2023-09-29T22:24:59.668 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.668 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.668 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.668 DEBUG:boto:Params: {} 2023-09-29T22:24:59.668 DEBUG:boto:Token: None 2023-09-29T22:24:59.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.669 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a46a9e5c165179-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.671 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.671 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.671 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.671 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.671 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.671 DEBUG:boto:Method: GET 2023-09-29T22:24:59.671 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.672 DEBUG:boto:Data: 2023-09-29T22:24:59.672 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.672 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.672 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.672 DEBUG:boto:Params: {} 2023-09-29T22:24:59.672 DEBUG:boto:Token: None 2023-09-29T22:24:59.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda18e1dbc6fe27f-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.675 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.675 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.675 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.675 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.676 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.676 DEBUG:boto:Data: 2023-09-29T22:24:59.676 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.676 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.676 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.676 DEBUG:boto:Params: {} 2023-09-29T22:24:59.676 DEBUG:boto:Token: None 2023-09-29T22:24:59.676 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA= 2023-09-29T22:24:59.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA=', 'Content-Length': '0'} 2023-09-29T22:24:59.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c5155aa2e590f6c-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.678 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.678 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.678 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.678 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.678 DEBUG:boto:Data: 2023-09-29T22:24:59.679 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.679 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.679 DEBUG:boto:Params: {} 2023-09-29T22:24:59.679 DEBUG:boto:Token: None 2023-09-29T22:24:59.679 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.679 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA= 2023-09-29T22:24:59.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA=', 'Content-Length': '0'} 2023-09-29T22:24:59.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047b15d28fbf238fc-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.682 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.682 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.682 DEBUG:boto:Method: GET 2023-09-29T22:24:59.682 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.682 DEBUG:boto:Data: 2023-09-29T22:24:59.682 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.682 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.682 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.682 DEBUG:boto:Params: {} 2023-09-29T22:24:59.683 DEBUG:boto:Token: None 2023-09-29T22:24:59.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.683 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI= 2023-09-29T22:24:59.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI=', 'Content-Length': '0'} 2023-09-29T22:24:59.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d9f8febf6c620b9-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.685 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.685 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.685 DEBUG:boto:Method: GET 2023-09-29T22:24:59.685 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.685 DEBUG:boto:Data: 2023-09-29T22:24:59.685 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.685 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.686 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.686 DEBUG:boto:Params: {} 2023-09-29T22:24:59.686 DEBUG:boto:Token: None 2023-09-29T22:24:59.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI= 2023-09-29T22:24:59.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI=', 'Content-Length': '0'} 2023-09-29T22:24:59.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c573f68d6ad7b42-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.688 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.688 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.688 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.688 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.688 DEBUG:boto:Method: GET 2023-09-29T22:24:59.688 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.689 DEBUG:boto:Data: 2023-09-29T22:24:59.689 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.689 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.689 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.689 DEBUG:boto:Params: {} 2023-09-29T22:24:59.689 DEBUG:boto:Token: None 2023-09-29T22:24:59.689 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.689 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c020b58c465e7407-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.691 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.692 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.692 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.692 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.692 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.692 DEBUG:boto:Method: GET 2023-09-29T22:24:59.692 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.692 DEBUG:boto:Data: 2023-09-29T22:24:59.693 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.693 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.693 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.693 DEBUG:boto:Params: {} 2023-09-29T22:24:59.693 DEBUG:boto:Token: None 2023-09-29T22:24:59.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.693 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7c04789c25b378-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.742 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.743 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.743 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.743 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.743 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.744 DEBUG:boto:Method: GET 2023-09-29T22:24:59.744 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.744 DEBUG:boto:Data: 2023-09-29T22:24:59.744 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.744 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.744 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.744 DEBUG:boto:Params: {} 2023-09-29T22:24:59.744 DEBUG:boto:Token: None 2023-09-29T22:24:59.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.745 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4= 2023-09-29T22:24:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4=', 'Content-Length': '0'} 2023-09-29T22:24:59.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000607e42950093294a-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.747 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.747 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.747 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.747 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.747 DEBUG:boto:Method: GET 2023-09-29T22:24:59.747 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.748 DEBUG:boto:Data: 2023-09-29T22:24:59.748 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.748 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.748 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.748 DEBUG:boto:Params: {} 2023-09-29T22:24:59.748 DEBUG:boto:Token: None 2023-09-29T22:24:59.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4= 2023-09-29T22:24:59.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4=', 'Content-Length': '0'} 2023-09-29T22:24:59.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f1f2074cc9b91fb-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.750 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.751 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.751 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.751 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.751 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.751 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.751 DEBUG:boto:Data: 2023-09-29T22:24:59.751 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.751 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.752 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.752 DEBUG:boto:Params: {} 2023-09-29T22:24:59.752 DEBUG:boto:Token: None 2023-09-29T22:24:59.752 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.752 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Juv1n5odBq6FBaMuwmkgrZagIbU= 2023-09-29T22:24:59.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Juv1n5odBq6FBaMuwmkgrZagIbU=', 'Content-Length': '0'} 2023-09-29T22:24:59.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010b3cc3fd57e1d48-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.754 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.754 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.754 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.754 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.755 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.755 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.755 DEBUG:boto:Data: 2023-09-29T22:24:59.755 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.755 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.755 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.755 DEBUG:boto:Params: {} 2023-09-29T22:24:59.755 DEBUG:boto:Token: None 2023-09-29T22:24:59.755 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.756 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Juv1n5odBq6FBaMuwmkgrZagIbU= 2023-09-29T22:24:59.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Juv1n5odBq6FBaMuwmkgrZagIbU=', 'Content-Length': '0'} 2023-09-29T22:24:59.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090e579f54fc041a2-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.758 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.758 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.758 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.758 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.759 DEBUG:boto:Method: GET 2023-09-29T22:24:59.759 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.759 DEBUG:boto:Data: 2023-09-29T22:24:59.759 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.759 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.759 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.759 DEBUG:boto:Params: {} 2023-09-29T22:24:59.759 DEBUG:boto:Token: None 2023-09-29T22:24:59.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4= 2023-09-29T22:24:59.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4=', 'Content-Length': '0'} 2023-09-29T22:24:59.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044bd082e99d5760b-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.762 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.762 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.762 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.762 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.763 DEBUG:boto:Method: GET 2023-09-29T22:24:59.763 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.763 DEBUG:boto:Data: 2023-09-29T22:24:59.763 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.763 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.763 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.763 DEBUG:boto:Params: {} 2023-09-29T22:24:59.763 DEBUG:boto:Token: None 2023-09-29T22:24:59.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:24:59.764 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4= 2023-09-29T22:24:59.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IVjBheEctiDSaW3PEbLuwkPwag4=', 'Content-Length': '0'} 2023-09-29T22:24:59.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad7eeda19a4d1455-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.766 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.766 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.766 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.766 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.766 DEBUG:boto:Method: GET 2023-09-29T22:24:59.766 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.766 DEBUG:boto:Data: 2023-09-29T22:24:59.767 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.767 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.767 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.767 DEBUG:boto:Params: {} 2023-09-29T22:24:59.767 DEBUG:boto:Token: None 2023-09-29T22:24:59.767 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.767 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c823306ddc1c11a-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.810 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.811 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.811 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.811 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.811 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.811 DEBUG:boto:Method: GET 2023-09-29T22:24:59.812 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.812 DEBUG:boto:Data: 2023-09-29T22:24:59.812 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.812 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.812 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.812 DEBUG:boto:Params: {} 2023-09-29T22:24:59.812 DEBUG:boto:Token: None 2023-09-29T22:24:59.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da9e55f437df53f-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.815 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.815 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.815 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.815 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.816 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.816 DEBUG:boto:Data: 2023-09-29T22:24:59.816 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.816 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.816 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.816 DEBUG:boto:Params: {} 2023-09-29T22:24:59.816 DEBUG:boto:Token: None 2023-09-29T22:24:59.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA= 2023-09-29T22:24:59.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA=', 'Content-Length': '0'} 2023-09-29T22:24:59.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000035d80ae51e08165-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.819 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.819 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.819 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.819 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.819 DEBUG:boto:Data: 2023-09-29T22:24:59.819 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.819 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.820 DEBUG:boto:Params: {} 2023-09-29T22:24:59.820 DEBUG:boto:Token: None 2023-09-29T22:24:59.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA= 2023-09-29T22:24:59.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dbu5PjMaLsoer+j9q61DgTpE8dA=', 'Content-Length': '0'} 2023-09-29T22:24:59.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060a847328d15177c-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.822 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.822 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.822 DEBUG:boto:Method: GET 2023-09-29T22:24:59.822 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.823 DEBUG:boto:Data: 2023-09-29T22:24:59.823 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.823 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.823 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.823 DEBUG:boto:Params: {} 2023-09-29T22:24:59.823 DEBUG:boto:Token: None 2023-09-29T22:24:59.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.823 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI= 2023-09-29T22:24:59.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI=', 'Content-Length': '0'} 2023-09-29T22:24:59.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ddb967bff7c5ab6-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.825 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.825 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.826 DEBUG:boto:Method: GET 2023-09-29T22:24:59.826 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:24:59.826 DEBUG:boto:Data: 2023-09-29T22:24:59.826 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.826 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.826 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.826 DEBUG:boto:Params: {} 2023-09-29T22:24:59.826 DEBUG:boto:Token: None 2023-09-29T22:24:59.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj 2023-09-29T22:24:59.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI= 2023-09-29T22:24:59.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Jja9nCF0Uq7TwmrD1zPLbrTCeRI=', 'Content-Length': '0'} 2023-09-29T22:24:59.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a27e8032342d5851-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.829 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.829 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.830 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.830 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.830 DEBUG:boto:Method: GET 2023-09-29T22:24:59.830 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.830 DEBUG:boto:Data: 2023-09-29T22:24:59.830 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.830 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:24:59.830 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.830 DEBUG:boto:Params: {} 2023-09-29T22:24:59.831 DEBUG:boto:Token: None 2023-09-29T22:24:59.831 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c61f33cb3c999ee-0065174ebb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.833 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.833 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.833 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.834 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.834 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.834 DEBUG:boto:Method: GET 2023-09-29T22:24:59.834 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.834 DEBUG:boto:Data: 2023-09-29T22:24:59.834 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.834 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.834 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.834 DEBUG:boto:Params: {} 2023-09-29T22:24:59.835 DEBUG:boto:Token: None 2023-09-29T22:24:59.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.835 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f570c55da9064e-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.878 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.879 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone2, test-zone1} 2023-09-29T22:24:59.879 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:24:59.879 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.879 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.879 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.879 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:59.879 DEBUG:boto:Data: 2023-09-29T22:24:59.880 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.880 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.880 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.880 DEBUG:boto:Params: {} 2023-09-29T22:24:59.880 DEBUG:boto:Token: None 2023-09-29T22:24:59.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c= 2023-09-29T22:24:59.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c=', 'Content-Length': '0'} 2023-09-29T22:24:59.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc8ff8baede909c8-0065174ebb-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.883 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.883 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.883 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.883 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:24:59.883 DEBUG:boto:Data: 2023-09-29T22:24:59.883 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.883 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.883 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.884 DEBUG:boto:Params: {} 2023-09-29T22:24:59.884 DEBUG:boto:Token: None 2023-09-29T22:24:59.884 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c= 2023-09-29T22:24:59.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YtAq9WXFJsL4Tjs1h1Wa+lUhi8c=', 'Content-Length': '0'} 2023-09-29T22:24:59.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017f84e37fdcd9e6c-0065174ebb-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:59.886 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.887 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.887 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.887 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.887 DEBUG:boto:Method: GET 2023-09-29T22:24:59.887 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.887 DEBUG:boto:Data: 2023-09-29T22:24:59.887 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.887 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.887 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.887 DEBUG:boto:Params: {} 2023-09-29T22:24:59.888 DEBUG:boto:Token: None 2023-09-29T22:24:59.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbb769b79e43a6c-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.890 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:59.891 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.891 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.891 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.891 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.892 DEBUG:boto:Method: GET 2023-09-29T22:24:59.892 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.892 DEBUG:boto:Data: 2023-09-29T22:24:59.892 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.892 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.892 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.892 DEBUG:boto:Params: {} 2023-09-29T22:24:59.892 DEBUG:boto:Token: None 2023-09-29T22:24:59.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.893 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eab58540bbb2709-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.895 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:24:59.896 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.896 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.896 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.896 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.896 DEBUG:boto:Method: GET 2023-09-29T22:24:59.896 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.896 DEBUG:boto:Data: 2023-09-29T22:24:59.896 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.896 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.897 DEBUG:boto:Params: {} 2023-09-29T22:24:59.897 DEBUG:boto:Token: None 2023-09-29T22:24:59.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b7558d03c844d1-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.942 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.943 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.943 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.943 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.943 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.943 DEBUG:boto:Method: GET 2023-09-29T22:24:59.943 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:24:59.943 DEBUG:boto:Data: 2023-09-29T22:24:59.943 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.944 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.944 DEBUG:boto:Params: {} 2023-09-29T22:24:59.944 DEBUG:boto:Token: None 2023-09-29T22:24:59.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/?versions 2023-09-29T22:24:59.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4= 2023-09-29T22:24:59.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8lrEJFGUe3PXt64kceQ/+AiB/C4=', 'Content-Length': '0'} 2023-09-29T22:24:59.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8aea1425f03030f-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.947 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.947 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.947 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.947 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.947 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.948 DEBUG:boto:Method: GET 2023-09-29T22:24:59.948 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.948 DEBUG:boto:Data: 2023-09-29T22:24:59.948 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.948 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.948 DEBUG:boto:Params: {} 2023-09-29T22:24:59.948 DEBUG:boto:Token: None 2023-09-29T22:24:59.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002059ca11cc5c0209-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.950 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.951 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.951 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.951 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.951 DEBUG:boto:Method: GET 2023-09-29T22:24:59.951 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.951 DEBUG:boto:Data: 2023-09-29T22:24:59.951 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.951 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.951 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.951 DEBUG:boto:Params: {} 2023-09-29T22:24:59.952 DEBUG:boto:Token: None 2023-09-29T22:24:59.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004f11e3bd02b766a-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.954 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.955 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.955 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.955 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.955 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.955 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.955 DEBUG:boto:Data: 2023-09-29T22:24:59.955 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.955 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.955 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.956 DEBUG:boto:Params: {} 2023-09-29T22:24:59.956 DEBUG:boto:Token: None 2023-09-29T22:24:59.956 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw= 2023-09-29T22:24:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw=', 'Content-Length': '0'} 2023-09-29T22:24:59.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000505ac905af1d86c-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.958 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.958 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.958 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.958 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.959 DEBUG:boto:Method: HEAD 2023-09-29T22:24:59.959 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.959 DEBUG:boto:Data: 2023-09-29T22:24:59.959 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.959 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.959 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.959 DEBUG:boto:Params: {} 2023-09-29T22:24:59.959 DEBUG:boto:Token: None 2023-09-29T22:24:59.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw= 2023-09-29T22:24:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ukzwpU8FJkVPNNxzLvKIQKsoZYw=', 'Content-Length': '0'} 2023-09-29T22:24:59.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045328d7fdf0672ed-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:24:59.962 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.962 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.962 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.962 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.962 DEBUG:boto:Method: GET 2023-09-29T22:24:59.962 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.963 DEBUG:boto:Data: 2023-09-29T22:24:59.963 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.963 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.963 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.963 DEBUG:boto:Params: {} 2023-09-29T22:24:59.963 DEBUG:boto:Token: None 2023-09-29T22:24:59.963 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.963 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7554bf08487c471-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.965 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:24:59.966 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:24:59.966 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.966 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.966 DEBUG:boto:Method: GET 2023-09-29T22:24:59.966 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.966 DEBUG:boto:Data: 2023-09-29T22:24:59.966 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.967 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.967 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.967 DEBUG:boto:Params: {} 2023-09-29T22:24:59.967 DEBUG:boto:Token: None 2023-09-29T22:24:59.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:24:59.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c= 2023-09-29T22:24:59.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HXGsvohQwnmwwJA3eqDhuR5ds2c=', 'Content-Length': '0'} 2023-09-29T22:24:59.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097d2bac9b881c3aa-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.969 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.969 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.969 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.970 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.970 DEBUG:boto:Method: GET 2023-09-29T22:24:59.970 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.970 DEBUG:boto:Data: 2023-09-29T22:24:59.970 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.970 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:24:59.970 DEBUG:boto:Port: 8001 2023-09-29T22:24:59.970 DEBUG:boto:Params: {} 2023-09-29T22:24:59.970 DEBUG:boto:Token: None 2023-09-29T22:24:59.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.971 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6abe8fec5ee9f46-0065174ebb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:59.973 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:59.974 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:24:59.974 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:24:59.974 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.974 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.974 DEBUG:boto:Method: GET 2023-09-29T22:24:59.974 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:24:59.974 DEBUG:boto:Data: 2023-09-29T22:24:59.975 DEBUG:boto:Headers: {} 2023-09-29T22:24:59.975 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:24:59.975 DEBUG:boto:Port: 8000 2023-09-29T22:24:59.975 DEBUG:boto:Params: {} 2023-09-29T22:24:59.975 DEBUG:boto:Token: None 2023-09-29T22:24:59.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:59 GMT /dmuhsu-20/ 2023-09-29T22:24:59.975 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o= 2023-09-29T22:24:59.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3C+o2++54X7b/h59GjMAnfIcm8o=', 'Content-Length': '0'} 2023-09-29T22:24:59.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da9c9bbb5514d7c-0065174ebb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.018 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.019 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.019 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.019 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.019 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.019 DEBUG:boto:Data: 2023-09-29T22:25:00.019 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.019 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.020 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.020 DEBUG:boto:Params: {} 2023-09-29T22:25:00.020 DEBUG:boto:Token: None 2023-09-29T22:25:00.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c4870e43eb6f4ce-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.022 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.022 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.022 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.023 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.023 DEBUG:boto:Data: 2023-09-29T22:25:00.023 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.023 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.023 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.023 DEBUG:boto:Params: {} 2023-09-29T22:25:00.023 DEBUG:boto:Token: None 2023-09-29T22:25:00.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.023 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000022b8546e5c7ddc0-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.026 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.026 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.026 DEBUG:boto:Method: GET 2023-09-29T22:25:00.026 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.026 DEBUG:boto:Data: 2023-09-29T22:25:00.026 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.026 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.027 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.027 DEBUG:boto:Params: {} 2023-09-29T22:25:00.027 DEBUG:boto:Token: None 2023-09-29T22:25:00.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.027 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b26101ec34091660-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.029 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.029 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.029 DEBUG:boto:Method: GET 2023-09-29T22:25:00.029 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.029 DEBUG:boto:Data: 2023-09-29T22:25:00.030 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.030 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.030 DEBUG:boto:Params: {} 2023-09-29T22:25:00.030 DEBUG:boto:Token: None 2023-09-29T22:25:00.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.030 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a73be39efccce1b0-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.032 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.032 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.032 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.033 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.033 DEBUG:boto:Method: GET 2023-09-29T22:25:00.033 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.033 DEBUG:boto:Data: 2023-09-29T22:25:00.033 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.033 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.033 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.033 DEBUG:boto:Params: {} 2023-09-29T22:25:00.033 DEBUG:boto:Token: None 2023-09-29T22:25:00.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ecec51bf6a41868-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.036 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.036 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.037 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.037 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.037 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.037 DEBUG:boto:Method: GET 2023-09-29T22:25:00.037 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.037 DEBUG:boto:Data: 2023-09-29T22:25:00.037 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.038 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.038 DEBUG:boto:Params: {} 2023-09-29T22:25:00.038 DEBUG:boto:Token: None 2023-09-29T22:25:00.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.038 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8807b0c341e139e-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.040 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.041 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.041 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.041 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.041 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.041 DEBUG:boto:Method: GET 2023-09-29T22:25:00.041 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.042 DEBUG:boto:Data: 2023-09-29T22:25:00.042 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.042 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.042 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.042 DEBUG:boto:Params: {} 2023-09-29T22:25:00.042 DEBUG:boto:Token: None 2023-09-29T22:25:00.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.042 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d860d9281b339f6-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.044 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.044 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.044 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.045 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.045 DEBUG:boto:Method: GET 2023-09-29T22:25:00.045 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.045 DEBUG:boto:Data: 2023-09-29T22:25:00.045 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.045 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.045 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.045 DEBUG:boto:Params: {} 2023-09-29T22:25:00.045 DEBUG:boto:Token: None 2023-09-29T22:25:00.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.046 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000541cb106fee31de5-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.048 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.048 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.048 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.048 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.048 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.048 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.048 DEBUG:boto:Data: 2023-09-29T22:25:00.049 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.049 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.049 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.049 DEBUG:boto:Params: {} 2023-09-29T22:25:00.049 DEBUG:boto:Token: None 2023-09-29T22:25:00.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.049 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM= 2023-09-29T22:25:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM=', 'Content-Length': '0'} 2023-09-29T22:25:00.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001061c0fe42045785-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.051 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.051 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.051 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.051 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.051 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.052 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.052 DEBUG:boto:Data: 2023-09-29T22:25:00.052 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.052 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.052 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.052 DEBUG:boto:Params: {} 2023-09-29T22:25:00.052 DEBUG:boto:Token: None 2023-09-29T22:25:00.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.053 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM= 2023-09-29T22:25:00.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM=', 'Content-Length': '0'} 2023-09-29T22:25:00.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df7b5090b3e2f2a3-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.055 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.055 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.055 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.055 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.055 DEBUG:boto:Method: GET 2023-09-29T22:25:00.055 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.056 DEBUG:boto:Data: 2023-09-29T22:25:00.056 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.056 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.056 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.056 DEBUG:boto:Params: {} 2023-09-29T22:25:00.056 DEBUG:boto:Token: None 2023-09-29T22:25:00.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e0faa34d4a20b75-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.058 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.058 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.059 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.059 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.059 DEBUG:boto:Method: GET 2023-09-29T22:25:00.059 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.059 DEBUG:boto:Data: 2023-09-29T22:25:00.059 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.059 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.059 DEBUG:boto:Params: {} 2023-09-29T22:25:00.060 DEBUG:boto:Token: None 2023-09-29T22:25:00.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e0585eea12274f1-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.062 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.062 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.062 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.062 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.062 DEBUG:boto:Method: GET 2023-09-29T22:25:00.062 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.063 DEBUG:boto:Data: 2023-09-29T22:25:00.063 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.063 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.063 DEBUG:boto:Params: {} 2023-09-29T22:25:00.063 DEBUG:boto:Token: None 2023-09-29T22:25:00.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.063 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006bc626f58d389a4-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.067 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.067 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.067 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.068 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.068 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.068 DEBUG:boto:Method: GET 2023-09-29T22:25:00.068 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.068 DEBUG:boto:Data: 2023-09-29T22:25:00.068 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.068 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.068 DEBUG:boto:Params: {} 2023-09-29T22:25:00.069 DEBUG:boto:Token: None 2023-09-29T22:25:00.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f0eaae8bbe8bd3-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.071 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.072 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.072 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.072 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.072 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.072 DEBUG:boto:Data: 2023-09-29T22:25:00.072 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.072 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.072 DEBUG:boto:Params: {} 2023-09-29T22:25:00.073 DEBUG:boto:Token: None 2023-09-29T22:25:00.073 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7da1f1a42fb5d87-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.075 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.075 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.076 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.076 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.076 DEBUG:boto:Data: 2023-09-29T22:25:00.076 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.076 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.076 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.076 DEBUG:boto:Params: {} 2023-09-29T22:25:00.076 DEBUG:boto:Token: None 2023-09-29T22:25:00.076 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.077 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a96dea6638bec8d-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.079 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.079 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.079 DEBUG:boto:Method: GET 2023-09-29T22:25:00.079 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.079 DEBUG:boto:Data: 2023-09-29T22:25:00.079 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.079 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.079 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.079 DEBUG:boto:Params: {} 2023-09-29T22:25:00.080 DEBUG:boto:Token: None 2023-09-29T22:25:00.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.080 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004eecb8d1f3023a41-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.082 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.082 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.082 DEBUG:boto:Method: GET 2023-09-29T22:25:00.082 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.082 DEBUG:boto:Data: 2023-09-29T22:25:00.082 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.082 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.083 DEBUG:boto:Params: {} 2023-09-29T22:25:00.083 DEBUG:boto:Token: None 2023-09-29T22:25:00.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.083 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2404683caad66b8-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.085 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.085 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.085 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.085 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.086 DEBUG:boto:Method: GET 2023-09-29T22:25:00.086 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.086 DEBUG:boto:Data: 2023-09-29T22:25:00.086 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.086 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.086 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.086 DEBUG:boto:Params: {} 2023-09-29T22:25:00.086 DEBUG:boto:Token: None 2023-09-29T22:25:00.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037f5a2885f2d91a7-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.135 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.135 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.135 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.135 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.136 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.136 DEBUG:boto:Method: GET 2023-09-29T22:25:00.136 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.136 DEBUG:boto:Data: 2023-09-29T22:25:00.136 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.136 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.136 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.136 DEBUG:boto:Params: {} 2023-09-29T22:25:00.136 DEBUG:boto:Token: None 2023-09-29T22:25:00.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.137 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090c5592755ad5bf5-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.141 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone1} 2023-09-29T22:25:00.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:25:00.142 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.142 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.142 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.142 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:25:00.142 DEBUG:boto:Data: 2023-09-29T22:25:00.142 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.142 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.142 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.142 DEBUG:boto:Params: {} 2023-09-29T22:25:00.143 DEBUG:boto:Token: None 2023-09-29T22:25:00.143 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.143 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WGhIpBEU1vl1Yt6yTXtjQM2bDKs= 2023-09-29T22:25:00.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WGhIpBEU1vl1Yt6yTXtjQM2bDKs=', 'Content-Length': '0'} 2023-09-29T22:25:00.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027eb56d277757c97-0065174ebc-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.145 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.145 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.146 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.146 DEBUG:boto:Path: /dmuhsu-20/ 2023-09-29T22:25:00.146 DEBUG:boto:Data: 2023-09-29T22:25:00.146 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.146 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.146 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.146 DEBUG:boto:Params: {} 2023-09-29T22:25:00.147 DEBUG:boto:Token: None 2023-09-29T22:25:00.147 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.147 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WGhIpBEU1vl1Yt6yTXtjQM2bDKs= 2023-09-29T22:25:00.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WGhIpBEU1vl1Yt6yTXtjQM2bDKs=', 'Content-Length': '0'} 2023-09-29T22:25:00.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc24f40feb961e83-0065174ebc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.149 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:25:00.150 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.150 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.150 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.150 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.150 DEBUG:boto:Method: GET 2023-09-29T22:25:00.150 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.150 DEBUG:boto:Data: 2023-09-29T22:25:00.151 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.151 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.151 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.151 DEBUG:boto:Params: {} 2023-09-29T22:25:00.151 DEBUG:boto:Token: None 2023-09-29T22:25:00.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/?versions 2023-09-29T22:25:00.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE= 2023-09-29T22:25:00.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE=', 'Content-Length': '0'} 2023-09-29T22:25:00.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c553a69cd8b581-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.154 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:00.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:00.154 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:25:00.154 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.155 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.155 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.155 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.155 DEBUG:boto:Method: GET 2023-09-29T22:25:00.155 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.155 DEBUG:boto:Data: 2023-09-29T22:25:00.155 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.155 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.155 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.156 DEBUG:boto:Params: {} 2023-09-29T22:25:00.156 DEBUG:boto:Token: None 2023-09-29T22:25:00.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/?versions 2023-09-29T22:25:00.156 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE= 2023-09-29T22:25:00.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE=', 'Content-Length': '0'} 2023-09-29T22:25:00.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad8b4f2073f8482-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.159 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:00.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:00.159 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.160 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.160 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.160 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.160 DEBUG:boto:Method: GET 2023-09-29T22:25:00.160 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.160 DEBUG:boto:Data: 2023-09-29T22:25:00.160 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.160 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.161 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.161 DEBUG:boto:Params: {} 2023-09-29T22:25:00.161 DEBUG:boto:Token: None 2023-09-29T22:25:00.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/?versions 2023-09-29T22:25:00.161 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE= 2023-09-29T22:25:00.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE=', 'Content-Length': '0'} 2023-09-29T22:25:00.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8222821b46cd07c-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.164 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.164 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.164 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.164 DEBUG:boto:path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.165 DEBUG:boto:auth_path=/dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.165 DEBUG:boto:Method: GET 2023-09-29T22:25:00.165 DEBUG:boto:Path: /dmuhsu-20/?versions&max-keys=999 2023-09-29T22:25:00.165 DEBUG:boto:Data: 2023-09-29T22:25:00.165 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.165 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.165 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.165 DEBUG:boto:Params: {} 2023-09-29T22:25:00.166 DEBUG:boto:Token: None 2023-09-29T22:25:00.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/?versions 2023-09-29T22:25:00.166 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE= 2023-09-29T22:25:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w5md6sVR6BvdTBBlTgoy5FfQVbE=', 'Content-Length': '0'} 2023-09-29T22:25:00.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009886119a31c58e27-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.210 DEBUG:boto:b'dmuhsu-20999falseobjnulltrue2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNv4xektc4xejzvQTQwXkSjB93oMIKJSfalse2023-09-29T22:23:47.197Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.211 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.211 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.211 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.212 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.212 DEBUG:boto:Method: GET 2023-09-29T22:25:00.212 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.212 DEBUG:boto:Data: 2023-09-29T22:25:00.212 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.212 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.212 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.212 DEBUG:boto:Params: {} 2023-09-29T22:25:00.212 DEBUG:boto:Token: None 2023-09-29T22:25:00.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.213 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc= 2023-09-29T22:25:00.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc=', 'Content-Length': '0'} 2023-09-29T22:25:00.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047b477fd9392b7ec-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.215 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.215 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.215 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.215 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.215 DEBUG:boto:Method: GET 2023-09-29T22:25:00.216 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.216 DEBUG:boto:Data: 2023-09-29T22:25:00.216 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.216 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.216 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.216 DEBUG:boto:Params: {} 2023-09-29T22:25:00.216 DEBUG:boto:Token: None 2023-09-29T22:25:00.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.217 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc= 2023-09-29T22:25:00.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc=', 'Content-Length': '0'} 2023-09-29T22:25:00.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d365c3812e78ef9d-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.219 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.219 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.219 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.219 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.219 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.219 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.219 DEBUG:boto:Data: 2023-09-29T22:25:00.220 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.220 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.220 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.220 DEBUG:boto:Params: {} 2023-09-29T22:25:00.220 DEBUG:boto:Token: None 2023-09-29T22:25:00.220 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9IJWn9fmDjeo+JdcGcx76Sc/P2s= 2023-09-29T22:25:00.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9IJWn9fmDjeo+JdcGcx76Sc/P2s=', 'Content-Length': '0'} 2023-09-29T22:25:00.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000298faff10a91dc97-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.222 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.222 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.223 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.223 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.223 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.223 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.223 DEBUG:boto:Data: 2023-09-29T22:25:00.223 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.223 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.224 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.224 DEBUG:boto:Params: {} 2023-09-29T22:25:00.224 DEBUG:boto:Token: None 2023-09-29T22:25:00.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.224 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9IJWn9fmDjeo+JdcGcx76Sc/P2s= 2023-09-29T22:25:00.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9IJWn9fmDjeo+JdcGcx76Sc/P2s=', 'Content-Length': '0'} 2023-09-29T22:25:00.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000cdf39f2f32e6818-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.227 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.227 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.227 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.227 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.227 DEBUG:boto:Method: GET 2023-09-29T22:25:00.227 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.227 DEBUG:boto:Data: 2023-09-29T22:25:00.228 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.228 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.228 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.228 DEBUG:boto:Params: {} 2023-09-29T22:25:00.228 DEBUG:boto:Token: None 2023-09-29T22:25:00.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc= 2023-09-29T22:25:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc=', 'Content-Length': '0'} 2023-09-29T22:25:00.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdf391bcabfef9ba-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.230 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.231 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.231 DEBUG:boto:path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.231 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.231 DEBUG:boto:Method: GET 2023-09-29T22:25:00.231 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.231 DEBUG:boto:Data: 2023-09-29T22:25:00.231 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.231 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.231 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.231 DEBUG:boto:Params: {} 2023-09-29T22:25:00.232 DEBUG:boto:Token: None 2023-09-29T22:25:00.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=null 2023-09-29T22:25:00.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc= 2023-09-29T22:25:00.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GZqiQGc2T2Erk+2c9aWTd9lwwjc=', 'Content-Length': '0'} 2023-09-29T22:25:00.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b214c86e1c94fda6-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.234 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.234 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.235 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.235 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.235 DEBUG:boto:Method: GET 2023-09-29T22:25:00.235 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.235 DEBUG:boto:Data: 2023-09-29T22:25:00.235 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.235 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.235 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.235 DEBUG:boto:Params: {} 2023-09-29T22:25:00.236 DEBUG:boto:Token: None 2023-09-29T22:25:00.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.236 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e7a54f2c8d63e56-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.282 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.283 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.283 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.283 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.283 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.283 DEBUG:boto:Method: GET 2023-09-29T22:25:00.283 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.283 DEBUG:boto:Data: 2023-09-29T22:25:00.284 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.284 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.284 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.284 DEBUG:boto:Params: {} 2023-09-29T22:25:00.284 DEBUG:boto:Token: None 2023-09-29T22:25:00.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecfea98fd26578db-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.287 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.287 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.287 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.288 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.288 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.288 DEBUG:boto:Data: 2023-09-29T22:25:00.288 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.288 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.288 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.288 DEBUG:boto:Params: {} 2023-09-29T22:25:00.288 DEBUG:boto:Token: None 2023-09-29T22:25:00.289 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.289 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008aef419d97beea14-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.291 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.291 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.291 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.291 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.291 DEBUG:boto:Data: 2023-09-29T22:25:00.291 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.292 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.292 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.292 DEBUG:boto:Params: {} 2023-09-29T22:25:00.292 DEBUG:boto:Token: None 2023-09-29T22:25:00.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065f07dcd07e5f510-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.295 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.295 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.295 DEBUG:boto:Method: GET 2023-09-29T22:25:00.295 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.295 DEBUG:boto:Data: 2023-09-29T22:25:00.295 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.295 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.295 DEBUG:boto:Params: {} 2023-09-29T22:25:00.296 DEBUG:boto:Token: None 2023-09-29T22:25:00.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001fd8aabe140138e2-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.298 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.298 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.298 DEBUG:boto:Method: GET 2023-09-29T22:25:00.298 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.298 DEBUG:boto:Data: 2023-09-29T22:25:00.299 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.299 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.299 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.299 DEBUG:boto:Params: {} 2023-09-29T22:25:00.299 DEBUG:boto:Token: None 2023-09-29T22:25:00.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eecbe386dc61a11a-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.302 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.302 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.302 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.302 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.302 DEBUG:boto:Method: GET 2023-09-29T22:25:00.302 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.302 DEBUG:boto:Data: 2023-09-29T22:25:00.302 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.303 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.303 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.303 DEBUG:boto:Params: {} 2023-09-29T22:25:00.303 DEBUG:boto:Token: None 2023-09-29T22:25:00.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.303 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749c47d0adee7545-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.306 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.306 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.306 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.307 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.307 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.307 DEBUG:boto:Method: GET 2023-09-29T22:25:00.307 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.307 DEBUG:boto:Data: 2023-09-29T22:25:00.307 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.307 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.307 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.307 DEBUG:boto:Params: {} 2023-09-29T22:25:00.308 DEBUG:boto:Token: None 2023-09-29T22:25:00.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.308 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759a37b36916b542-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.355 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.355 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.356 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.356 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.356 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.356 DEBUG:boto:Method: GET 2023-09-29T22:25:00.356 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.356 DEBUG:boto:Data: 2023-09-29T22:25:00.356 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.356 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.357 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.357 DEBUG:boto:Params: {} 2023-09-29T22:25:00.357 DEBUG:boto:Token: None 2023-09-29T22:25:00.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000294e05275d316195-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.360 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.360 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.360 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.360 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.360 DEBUG:boto:Method: GET 2023-09-29T22:25:00.360 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.360 DEBUG:boto:Data: 2023-09-29T22:25:00.361 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.361 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.361 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.361 DEBUG:boto:Params: {} 2023-09-29T22:25:00.361 DEBUG:boto:Token: None 2023-09-29T22:25:00.361 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.361 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab44cf92b81eb4b6-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.364 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.364 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.364 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.364 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.365 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.365 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.365 DEBUG:boto:Data: 2023-09-29T22:25:00.365 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.365 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.365 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.365 DEBUG:boto:Params: {} 2023-09-29T22:25:00.366 DEBUG:boto:Token: None 2023-09-29T22:25:00.366 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.366 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM= 2023-09-29T22:25:00.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM=', 'Content-Length': '0'} 2023-09-29T22:25:00.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000287b7bd6e4117df8-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.368 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.368 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.368 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.369 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.369 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.369 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.369 DEBUG:boto:Data: 2023-09-29T22:25:00.369 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.369 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.369 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.369 DEBUG:boto:Params: {} 2023-09-29T22:25:00.370 DEBUG:boto:Token: None 2023-09-29T22:25:00.370 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.370 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM= 2023-09-29T22:25:00.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PCuemd77zHle5+sXIRLDV3wV7HM=', 'Content-Length': '0'} 2023-09-29T22:25:00.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c440bc51db2ed3d-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.373 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.373 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.373 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.373 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.373 DEBUG:boto:Method: GET 2023-09-29T22:25:00.373 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.373 DEBUG:boto:Data: 2023-09-29T22:25:00.373 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.374 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.374 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.374 DEBUG:boto:Params: {} 2023-09-29T22:25:00.374 DEBUG:boto:Token: None 2023-09-29T22:25:00.374 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.375 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9779eeaf2e99012-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.377 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.377 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.378 DEBUG:boto:path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.378 DEBUG:boto:auth_path=/dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.378 DEBUG:boto:Method: GET 2023-09-29T22:25:00.378 DEBUG:boto:Path: /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.378 DEBUG:boto:Data: 2023-09-29T22:25:00.379 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.379 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.379 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.379 DEBUG:boto:Params: {} 2023-09-29T22:25:00.380 DEBUG:boto:Token: None 2023-09-29T22:25:00.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj?versionId=Nv4xektc4xejzvQTQwXkSjB93oMIKJS 2023-09-29T22:25:00.380 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo= 2023-09-29T22:25:00.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7aO+mc1i4yVz6zPtx/IOWk745jo=', 'Content-Length': '0'} 2023-09-29T22:25:00.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:47 GMT'), ('x-amz-version-id', 'Nv4xektc4xejzvQTQwXkSjB93oMIKJS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e609fbc79abd9c43-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.383 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.383 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.383 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.383 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.383 DEBUG:boto:Method: GET 2023-09-29T22:25:00.384 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.384 DEBUG:boto:Data: 2023-09-29T22:25:00.384 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.384 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.384 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.384 DEBUG:boto:Params: {} 2023-09-29T22:25:00.384 DEBUG:boto:Token: None 2023-09-29T22:25:00.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.385 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad77a6df5efd5a3a-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.430 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.431 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.431 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.431 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.431 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.431 DEBUG:boto:Method: GET 2023-09-29T22:25:00.431 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.432 DEBUG:boto:Data: 2023-09-29T22:25:00.432 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.432 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.432 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.432 DEBUG:boto:Params: {} 2023-09-29T22:25:00.432 DEBUG:boto:Token: None 2023-09-29T22:25:00.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.432 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066f66191572718c4-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.478 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.479 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.479 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.479 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.479 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.479 DEBUG:boto:Data: 2023-09-29T22:25:00.479 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.479 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.480 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.480 DEBUG:boto:Params: {} 2023-09-29T22:25:00.480 DEBUG:boto:Token: None 2023-09-29T22:25:00.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000122123c632e2fd7a-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.482 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.482 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.483 DEBUG:boto:Method: HEAD 2023-09-29T22:25:00.483 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.483 DEBUG:boto:Data: 2023-09-29T22:25:00.483 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.483 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.483 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.483 DEBUG:boto:Params: {} 2023-09-29T22:25:00.483 DEBUG:boto:Token: None 2023-09-29T22:25:00.483 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.484 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0= 2023-09-29T22:25:00.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BUvkGwiAT8i9eS7fCKXUpqZ9se0=', 'Content-Length': '0'} 2023-09-29T22:25:00.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001862b2098da2ade8-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:00.486 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.486 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.486 DEBUG:boto:Method: GET 2023-09-29T22:25:00.486 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.486 DEBUG:boto:Data: 2023-09-29T22:25:00.486 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.487 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.487 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.487 DEBUG:boto:Params: {} 2023-09-29T22:25:00.487 DEBUG:boto:Token: None 2023-09-29T22:25:00.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.487 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.489 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b92718cd0700dd5d-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.489 DEBUG:boto:path=/dmuhsu-20/obj 2023-09-29T22:25:00.490 DEBUG:boto:auth_path=/dmuhsu-20/obj 2023-09-29T22:25:00.490 DEBUG:boto:Method: GET 2023-09-29T22:25:00.490 DEBUG:boto:Path: /dmuhsu-20/obj 2023-09-29T22:25:00.490 DEBUG:boto:Data: 2023-09-29T22:25:00.490 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.490 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.490 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.490 DEBUG:boto:Params: {} 2023-09-29T22:25:00.490 DEBUG:boto:Token: None 2023-09-29T22:25:00.491 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/obj 2023-09-29T22:25:00.491 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo= 2023-09-29T22:25:00.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D7gTPr3P2fS9bqMkEZM8jjWfJVo=', 'Content-Length': '0'} 2023-09-29T22:25:00.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000151e43ab9760a962-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.493 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.493 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.493 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.493 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.493 DEBUG:boto:Method: GET 2023-09-29T22:25:00.493 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.493 DEBUG:boto:Data: 2023-09-29T22:25:00.494 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.494 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.494 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.494 DEBUG:boto:Params: {} 2023-09-29T22:25:00.494 DEBUG:boto:Token: None 2023-09-29T22:25:00.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.494 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a96bf419373936b-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.497 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.498 DEBUG:boto:path=/dmuhsu-20/ 2023-09-29T22:25:00.498 DEBUG:boto:auth_path=/dmuhsu-20/ 2023-09-29T22:25:00.498 DEBUG:boto:path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.498 DEBUG:boto:auth_path=/dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.498 DEBUG:boto:Method: GET 2023-09-29T22:25:00.498 DEBUG:boto:Path: /dmuhsu-20/?prefix=obj 2023-09-29T22:25:00.498 DEBUG:boto:Data: 2023-09-29T22:25:00.498 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.498 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.499 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.499 DEBUG:boto:Params: {} 2023-09-29T22:25:00.499 DEBUG:boto:Token: None 2023-09-29T22:25:00.499 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-20/ 2023-09-29T22:25:00.499 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M= 2023-09-29T22:25:00.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zEEkEJcINtpIzNqQoR+S+IGz/6M=', 'Content-Length': '0'} 2023-09-29T22:25:00.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776ee15fc3630880-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.502 DEBUG:boto:b'dmuhsu-20obj1000falseobj2023-09-29T22:24:23.921Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:00.502 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-20 zones={test-zone3, test-zone2} 2023-09-29T22:25:00.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-09-29T22:25:00.503 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-21 2023-09-29T22:25:00.504 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:00.504 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:00.504 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.504 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:00.504 DEBUG:boto:Data: 2023-09-29T22:25:00.504 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.504 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.505 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.505 DEBUG:boto:Params: {} 2023-09-29T22:25:00.505 DEBUG:boto:Token: None 2023-09-29T22:25:00.505 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-21/ 2023-09-29T22:25:00.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JJkSi+GPn/+Z2T8TlWbxf4FwBsA= 2023-09-29T22:25:00.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JJkSi+GPn/+Z2T8TlWbxf4FwBsA=', 'Content-Length': '0'} 2023-09-29T22:25:00.520 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090edef3d2635bed5-0065174ebc-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.520 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-22 2023-09-29T22:25:00.520 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:25:00.521 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:25:00.521 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.521 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:25:00.521 DEBUG:boto:Data: 2023-09-29T22:25:00.521 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.521 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.521 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.521 DEBUG:boto:Params: {} 2023-09-29T22:25:00.521 DEBUG:boto:Token: None 2023-09-29T22:25:00.522 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-22/ 2023-09-29T22:25:00.522 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ClubnCt086LuZm7iLm9ZQliua20= 2023-09-29T22:25:00.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ClubnCt086LuZm7iLm9ZQliua20=', 'Content-Length': '0'} 2023-09-29T22:25:00.550 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df55fefc905aa1b5-0065174ebc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.621 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-23 2023-09-29T22:25:00.621 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:25:00.621 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:25:00.621 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.622 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:25:00.622 DEBUG:boto:Data: 2023-09-29T22:25:00.622 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.622 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.622 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.622 DEBUG:boto:Params: {} 2023-09-29T22:25:00.623 DEBUG:boto:Token: None 2023-09-29T22:25:00.623 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-23/ 2023-09-29T22:25:00.623 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y5X6mmcTEuDflHAGoIf64yoSwJk= 2023-09-29T22:25:00.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y5X6mmcTEuDflHAGoIf64yoSwJk=', 'Content-Length': '0'} 2023-09-29T22:25:00.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ad79f4778d6f488-0065174ebc-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.648 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:00.648 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:00.648 DEBUG:boto:path=/dmuhsu-21/?versioning 2023-09-29T22:25:00.648 DEBUG:boto:auth_path=/dmuhsu-21/?versioning 2023-09-29T22:25:00.648 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.648 DEBUG:boto:Path: /dmuhsu-21/?versioning 2023-09-29T22:25:00.648 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:25:00.648 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.648 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:00.649 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.649 DEBUG:boto:Params: {} 2023-09-29T22:25:00.649 DEBUG:boto:Token: None 2023-09-29T22:25:00.649 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-21/?versioning 2023-09-29T22:25:00.649 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Odfkrf2T+4tNPyTkLp/2oJ+Pvp4= 2023-09-29T22:25:00.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Odfkrf2T+4tNPyTkLp/2oJ+Pvp4=', 'Content-Length': '228'} 2023-09-29T22:25:00.654 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f15875a787ca829c-0065174ebc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.654 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:25:00.654 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:25:00.655 DEBUG:boto:path=/dmuhsu-22/?versioning 2023-09-29T22:25:00.655 DEBUG:boto:auth_path=/dmuhsu-22/?versioning 2023-09-29T22:25:00.655 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.655 DEBUG:boto:Path: /dmuhsu-22/?versioning 2023-09-29T22:25:00.655 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:25:00.655 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.655 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:00.655 DEBUG:boto:Port: 8000 2023-09-29T22:25:00.655 DEBUG:boto:Params: {} 2023-09-29T22:25:00.656 DEBUG:boto:Token: None 2023-09-29T22:25:00.656 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-22/?versioning 2023-09-29T22:25:00.656 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H2tU9iS2ClFzDWg+Yp4PoGNlH9s= 2023-09-29T22:25:00.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H2tU9iS2ClFzDWg+Yp4PoGNlH9s=', 'Content-Length': '228'} 2023-09-29T22:25:00.664 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a6c723c41cf6ebf-0065174ebc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.664 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:25:00.664 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:25:00.664 DEBUG:boto:path=/dmuhsu-23/?versioning 2023-09-29T22:25:00.664 DEBUG:boto:auth_path=/dmuhsu-23/?versioning 2023-09-29T22:25:00.664 DEBUG:boto:Method: PUT 2023-09-29T22:25:00.665 DEBUG:boto:Path: /dmuhsu-23/?versioning 2023-09-29T22:25:00.665 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:25:00.665 DEBUG:boto:Headers: {} 2023-09-29T22:25:00.665 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:00.665 DEBUG:boto:Port: 8001 2023-09-29T22:25:00.665 DEBUG:boto:Params: {} 2023-09-29T22:25:00.665 DEBUG:boto:Token: None 2023-09-29T22:25:00.665 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:00 GMT /dmuhsu-23/?versioning 2023-09-29T22:25:00.665 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:moK+k/1nQvke0m3iGBD0uCMdwes= 2023-09-29T22:25:00.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:moK+k/1nQvke0m3iGBD0uCMdwes=', 'Content-Length': '228'} 2023-09-29T22:25:00.672 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000923743208b556a6a-0065174ebc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:00.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:00.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:00.673 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:00.725 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:00.725 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:00.770 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:25:00.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026300.671472_150.1", 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:25:00.671472Z" 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:00.783 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:25:00.784 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:00.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:25:00.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:00.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:00.785 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:00.834 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:00.835 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:00.866 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:00.866+0000 7f49dca13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:00.867 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:00.867+0000 7f49dca13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:00.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:00.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026300.517098_135.1", 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:00.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:25:00.517098Z", 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:00.885 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:00.886 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:25:00Z" 2023-09-29T22:25:00.887 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:25:00.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026300.517098_135.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:00.517098Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:00Z" } 2023-09-29T22:25:00.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026300.517098_135.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:00.517098Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:25:00.888 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:00.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.517098_135.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:00.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026300.671472_150.1 target=1_1696026300.517098_135.1 2023-09-29T22:25:05.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:05.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:05.890 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:05.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:05.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:05.974 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:05.974+0000 7fe637137880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:05.975 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:05.975+0000 7fe637137880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:05.987 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:25:05.987 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:25:05.988 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:05.990 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026300.516104_134.1", 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:05.991 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:25:00.516104Z", 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:05.992 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:25:05.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:05.994 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:25:05Z" 2023-09-29T22:25:05.994 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:25:05.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026300.516104_134.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:00.516104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:05Z" } 2023-09-29T22:25:05.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026300.516104_134.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:00.516104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:25:05.995 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:05.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.516104_134.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:05.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026300.671472_150.1 target=1_1696026300.516104_134.1 2023-09-29T22:25:10.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:10.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:10.996 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:11.049 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:11.049 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:11.080 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:11.080+0000 7f33c3133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:11.081 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:11.081+0000 7f33c3133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026300.671472_150.1", 2023-09-29T22:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:25:00.671472Z", 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:25:11.098 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:25:11.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:25:11Z" 2023-09-29T22:25:11.100 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:25:11.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026300.671472_150.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:00.671472Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:11Z" } 2023-09-29T22:25:11.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026300.671472_150.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:00.671472Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:25:11.101 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:11.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:11.101 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:25:11.102 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:25:11.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:11.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:11.102 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:11.154 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:11.154 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:11.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:11.189+0000 7fb75bba2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:11.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:11.190+0000 7fb75bba2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:11.202 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:25:11.202 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:25:11.203 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:11.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026300.671472_150.1", 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:25:00.671472Z", 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:25:11.206 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:25:11.207 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:25:11Z" 2023-09-29T22:25:11.208 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:25:11.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026300.671472_150.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:00.671472Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:11Z" } 2023-09-29T22:25:11.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026300.671472_150.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:00.671472Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:25:11.210 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:11.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026300.671472_150.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:25:11.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:25:11.210 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:11.210 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:11.210 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.210 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:11.210 DEBUG:boto:Data: 2023-09-29T22:25:11.211 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.211 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:11.211 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.211 DEBUG:boto:Params: {} 2023-09-29T22:25:11.211 DEBUG:boto:Token: None 2023-09-29T22:25:11.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-21/ 2023-09-29T22:25:11.211 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M3vpGapewT9MzquBp6SSieS+ZL0= 2023-09-29T22:25:11.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M3vpGapewT9MzquBp6SSieS+ZL0=', 'Content-Length': '0'} 2023-09-29T22:25:11.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006308c99a9fb603fc-0065174ec7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.214 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:11.214 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:11.214 DEBUG:boto:Method: PUT 2023-09-29T22:25:11.214 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:11.214 DEBUG:boto:Data: 2023-09-29T22:25:11.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:25:11.214 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:11.215 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.215 DEBUG:boto:Params: {} 2023-09-29T22:25:11.215 DEBUG:boto:Token: None 2023-09-29T22:25:11.215 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-21/obj 2023-09-29T22:25:11.215 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:83yygbhI23A3gnPnzARhFX1YDG8= 2023-09-29T22:25:11.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:83yygbhI23A3gnPnzARhFX1YDG8='} 2023-09-29T22:25:11.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-amz-request-id', 'tx00000e39aa2106ba4c438-0065174ec7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.225 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:11.225 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:11.225 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.226 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:11.226 DEBUG:boto:Data: 2023-09-29T22:25:11.226 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.226 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:11.226 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.226 DEBUG:boto:Params: {} 2023-09-29T22:25:11.226 DEBUG:boto:Token: None 2023-09-29T22:25:11.226 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-21/ 2023-09-29T22:25:11.226 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M3vpGapewT9MzquBp6SSieS+ZL0= 2023-09-29T22:25:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M3vpGapewT9MzquBp6SSieS+ZL0=', 'Content-Length': '0'} 2023-09-29T22:25:11.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005958b71739575589-0065174ec7-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.229 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:11.229 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:11.229 DEBUG:boto:Method: DELETE 2023-09-29T22:25:11.229 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:11.229 DEBUG:boto:Data: 2023-09-29T22:25:11.229 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.229 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:11.229 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.229 DEBUG:boto:Params: {} 2023-09-29T22:25:11.229 DEBUG:boto:Token: None 2023-09-29T22:25:11.230 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-21/obj 2023-09-29T22:25:11.230 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9lCL0uiciepSfZxdrxUkXif4uGc= 2023-09-29T22:25:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9lCL0uiciepSfZxdrxUkXif4uGc=', 'Content-Length': '0'} 2023-09-29T22:25:11.235 DEBUG:boto:Response headers: [('x-amz-version-id', 'MoR.hd6ln11K0fMmlF.jLnAjvI94vkA'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e7645610bb0f11c0-0065174ec7-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.235 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:25:11.236 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:25:11.236 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.236 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:25:11.236 DEBUG:boto:Data: 2023-09-29T22:25:11.236 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.236 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:11.236 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.236 DEBUG:boto:Params: {} 2023-09-29T22:25:11.236 DEBUG:boto:Token: None 2023-09-29T22:25:11.236 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-22/ 2023-09-29T22:25:11.237 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SO+jxPRYpl90YpxtopxeCCKOCwY= 2023-09-29T22:25:11.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SO+jxPRYpl90YpxtopxeCCKOCwY=', 'Content-Length': '0'} 2023-09-29T22:25:11.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5b8c979d68716fa-0065174ec7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.239 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:25:11.239 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:25:11.239 DEBUG:boto:Method: PUT 2023-09-29T22:25:11.239 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:25:11.239 DEBUG:boto:Data: 2023-09-29T22:25:11.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:25:11.240 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:11.240 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.240 DEBUG:boto:Params: {} 2023-09-29T22:25:11.240 DEBUG:boto:Token: None 2023-09-29T22:25:11.240 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-22/obj 2023-09-29T22:25:11.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z/EE+IDb15inPKLIrEsw71K5M/Q= 2023-09-29T22:25:11.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z/EE+IDb15inPKLIrEsw71K5M/Q='} 2023-09-29T22:25:11.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-amz-request-id', 'tx00000fb9ee190095db7db-0065174ec7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.252 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:25:11.252 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:25:11.252 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.252 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:25:11.252 DEBUG:boto:Data: 2023-09-29T22:25:11.252 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.252 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:11.252 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.253 DEBUG:boto:Params: {} 2023-09-29T22:25:11.253 DEBUG:boto:Token: None 2023-09-29T22:25:11.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-22/ 2023-09-29T22:25:11.253 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SO+jxPRYpl90YpxtopxeCCKOCwY= 2023-09-29T22:25:11.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SO+jxPRYpl90YpxtopxeCCKOCwY=', 'Content-Length': '0'} 2023-09-29T22:25:11.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e1fe430e6fe5fe5-0065174ec7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.255 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:25:11.255 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:25:11.255 DEBUG:boto:Method: DELETE 2023-09-29T22:25:11.255 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:25:11.255 DEBUG:boto:Data: 2023-09-29T22:25:11.255 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.256 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:11.256 DEBUG:boto:Port: 8000 2023-09-29T22:25:11.256 DEBUG:boto:Params: {} 2023-09-29T22:25:11.256 DEBUG:boto:Token: None 2023-09-29T22:25:11.256 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-22/obj 2023-09-29T22:25:11.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:w1SvMk3Ejw/d0ggO8Ga3zI0oqcM= 2023-09-29T22:25:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:w1SvMk3Ejw/d0ggO8Ga3zI0oqcM=', 'Content-Length': '0'} 2023-09-29T22:25:11.265 DEBUG:boto:Response headers: [('x-amz-version-id', 'mTCV5nmmcQfHxVdLZT90LbuEfsXE.Fr'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000003cee197df3df9a37-0065174ec7-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.265 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:25:11.265 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:25:11.266 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.266 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:25:11.266 DEBUG:boto:Data: 2023-09-29T22:25:11.266 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.266 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:11.266 DEBUG:boto:Port: 8001 2023-09-29T22:25:11.266 DEBUG:boto:Params: {} 2023-09-29T22:25:11.266 DEBUG:boto:Token: None 2023-09-29T22:25:11.266 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-23/ 2023-09-29T22:25:11.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0qihB7UZt1EsVbXgzuXSoAGheV8= 2023-09-29T22:25:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0qihB7UZt1EsVbXgzuXSoAGheV8=', 'Content-Length': '0'} 2023-09-29T22:25:11.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d62340b3704ffa86-0065174ec7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.269 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:25:11.270 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:25:11.270 DEBUG:boto:Method: PUT 2023-09-29T22:25:11.270 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:25:11.270 DEBUG:boto:Data: 2023-09-29T22:25:11.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:25:11.270 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:11.270 DEBUG:boto:Port: 8001 2023-09-29T22:25:11.270 DEBUG:boto:Params: {} 2023-09-29T22:25:11.270 DEBUG:boto:Token: None 2023-09-29T22:25:11.271 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-23/obj 2023-09-29T22:25:11.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NsmhEGOrRHMfB4din1JyWfWlhg8= 2023-09-29T22:25:11.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NsmhEGOrRHMfB4din1JyWfWlhg8='} 2023-09-29T22:25:11.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-amz-request-id', 'tx00000624b67567faecf92-0065174ec7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.283 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:25:11.283 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:25:11.283 DEBUG:boto:Method: HEAD 2023-09-29T22:25:11.283 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:25:11.283 DEBUG:boto:Data: 2023-09-29T22:25:11.284 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.284 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:11.284 DEBUG:boto:Port: 8001 2023-09-29T22:25:11.284 DEBUG:boto:Params: {} 2023-09-29T22:25:11.284 DEBUG:boto:Token: None 2023-09-29T22:25:11.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-23/ 2023-09-29T22:25:11.284 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0qihB7UZt1EsVbXgzuXSoAGheV8= 2023-09-29T22:25:11.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0qihB7UZt1EsVbXgzuXSoAGheV8=', 'Content-Length': '0'} 2023-09-29T22:25:11.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc2d1f84fc2ef9a7-0065174ec7-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.286 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:25:11.286 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:25:11.286 DEBUG:boto:Method: DELETE 2023-09-29T22:25:11.287 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:25:11.287 DEBUG:boto:Data: 2023-09-29T22:25:11.287 DEBUG:boto:Headers: {} 2023-09-29T22:25:11.287 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:11.287 DEBUG:boto:Port: 8001 2023-09-29T22:25:11.287 DEBUG:boto:Params: {} 2023-09-29T22:25:11.287 DEBUG:boto:Token: None 2023-09-29T22:25:11.287 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:25:11 GMT /dmuhsu-23/obj 2023-09-29T22:25:11.287 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FadVwRS8vodsoA/oabknRXy/N2o= 2023-09-29T22:25:11.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:11 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FadVwRS8vodsoA/oabknRXy/N2o=', 'Content-Length': '0'} 2023-09-29T22:25:11.293 DEBUG:boto:Response headers: [('x-amz-version-id', '9vq1bAE7CzFqma6nyrYjW5tzcbokSYo'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000c59b7783ad1c31b1-0065174ec7-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:11.294 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-21 2023-09-29T22:25:11.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:11.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:11.294 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:11.347 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:11.347 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:11.381 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:11.381+0000 7f433151f880 1 waiting to reach incremental sync.. 2023-09-29T22:25:16.383 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:16.382+0000 7f433151f880 1 bucket sync caught up with source: 2023-09-29T22:25:16.383 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.14.12, , ] 2023-09-29T22:25:16.383 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2023-09-29T22:25:16.383 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:16.382+0000 7f433151f880 0 bucket checkpoint complete 2023-09-29T22:25:16.399 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-21 2023-09-29T22:25:16.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:16.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:16.399 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:16.459 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:16.459 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:16.495 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:16.495+0000 7f4500015880 1 waiting to reach incremental sync.. 2023-09-29T22:25:21.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.496+0000 7f4500015880 1 bucket sync caught up with source: 2023-09-29T22:25:21.497 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.14.12, , ] 2023-09-29T22:25:21.497 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2023-09-29T22:25:21.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.496+0000 7f4500015880 0 bucket checkpoint complete 2023-09-29T22:25:21.513 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-21 2023-09-29T22:25:21.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.514 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:21.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:21.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:21.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.607+0000 7f49fb175880 1 bucket sync caught up with source: 2023-09-29T22:25:21.608 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.111.6, , ] 2023-09-29T22:25:21.608 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.111.6, , ] 2023-09-29T22:25:21.608 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.607+0000 7f49fb175880 0 bucket checkpoint complete 2023-09-29T22:25:21.620 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-21 2023-09-29T22:25:21.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.620 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:21.673 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:21.673 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:21.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.706+0000 7f09ad444880 1 bucket sync caught up with source: 2023-09-29T22:25:21.706 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.111.6, , ] 2023-09-29T22:25:21.706 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.111.6, , ] 2023-09-29T22:25:21.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.706+0000 7f09ad444880 0 bucket checkpoint complete 2023-09-29T22:25:21.718 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-21 2023-09-29T22:25:21.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:21.719 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:21.771 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:21.771 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:21.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:21.805+0000 7f62fec30880 1 waiting to reach incremental sync.. 2023-09-29T22:25:26.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:26.806+0000 7f62fec30880 1 waiting to reach incremental sync.. 2023-09-29T22:25:31.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:31.806+0000 7f62fec30880 1 waiting to reach incremental sync.. 2023-09-29T22:25:36.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:36.806+0000 7f62fec30880 1 waiting to reach incremental sync.. 2023-09-29T22:25:41.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:41.807+0000 7f62fec30880 1 bucket sync caught up with source: 2023-09-29T22:25:41.809 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.107.6, , ] 2023-09-29T22:25:41.809 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.107.6, , ] 2023-09-29T22:25:41.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:41.807+0000 7f62fec30880 0 bucket checkpoint complete 2023-09-29T22:25:41.826 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-21 2023-09-29T22:25:41.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:41.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:41.828 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:41.888 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:41.889 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:41.925 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:41.925+0000 7f353f410880 1 bucket sync caught up with source: 2023-09-29T22:25:41.925 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.107.6, , ] 2023-09-29T22:25:41.925 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.107.6, , ] 2023-09-29T22:25:41.926 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:41.925+0000 7f353f410880 0 bucket checkpoint complete 2023-09-29T22:25:41.938 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-21 zones={test-zone2, test-zone1} 2023-09-29T22:25:41.939 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.939 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.939 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.939 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:41.939 DEBUG:boto:Data: 2023-09-29T22:25:41.939 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.939 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.940 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.940 DEBUG:boto:Params: {} 2023-09-29T22:25:41.940 DEBUG:boto:Token: None 2023-09-29T22:25:41.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/ 2023-09-29T22:25:41.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SbhwdA4wEZZL6OVO99aTlhh5/EA= 2023-09-29T22:25:41.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SbhwdA4wEZZL6OVO99aTlhh5/EA=', 'Content-Length': '0'} 2023-09-29T22:25:41.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008835a4b8d2ce5e16-0065174ee5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.943 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.943 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.943 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.943 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:41.943 DEBUG:boto:Data: 2023-09-29T22:25:41.944 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.944 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.944 DEBUG:boto:Params: {} 2023-09-29T22:25:41.944 DEBUG:boto:Token: None 2023-09-29T22:25:41.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/ 2023-09-29T22:25:41.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SbhwdA4wEZZL6OVO99aTlhh5/EA= 2023-09-29T22:25:41.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SbhwdA4wEZZL6OVO99aTlhh5/EA=', 'Content-Length': '0'} 2023-09-29T22:25:41.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bfec7d637ea4eef-0065174ee5-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:25:41.948 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.948 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.948 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.948 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.948 DEBUG:boto:Method: GET 2023-09-29T22:25:41.948 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.948 DEBUG:boto:Data: 2023-09-29T22:25:41.948 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.948 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.949 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.949 DEBUG:boto:Params: {} 2023-09-29T22:25:41.949 DEBUG:boto:Token: None 2023-09-29T22:25:41.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/?versions 2023-09-29T22:25:41.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE= 2023-09-29T22:25:41.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE=', 'Content-Length': '0'} 2023-09-29T22:25:41.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012424cd0b4793a25-0065174ee5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.951 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:41.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:41.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:41.953 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:25:41.953 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.953 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.953 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.953 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.953 DEBUG:boto:Method: GET 2023-09-29T22:25:41.953 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.953 DEBUG:boto:Data: 2023-09-29T22:25:41.953 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.953 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.953 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.954 DEBUG:boto:Params: {} 2023-09-29T22:25:41.954 DEBUG:boto:Token: None 2023-09-29T22:25:41.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/?versions 2023-09-29T22:25:41.954 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE= 2023-09-29T22:25:41.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE=', 'Content-Length': '0'} 2023-09-29T22:25:41.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004615ee39fb3fd940-0065174ee5-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.957 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:41.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:41.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:41.958 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.958 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.958 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.958 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.959 DEBUG:boto:Method: GET 2023-09-29T22:25:41.959 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.959 DEBUG:boto:Data: 2023-09-29T22:25:41.959 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.959 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.959 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.959 DEBUG:boto:Params: {} 2023-09-29T22:25:41.959 DEBUG:boto:Token: None 2023-09-29T22:25:41.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/?versions 2023-09-29T22:25:41.960 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE= 2023-09-29T22:25:41.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE=', 'Content-Length': '0'} 2023-09-29T22:25:41.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f4983af09d46d1-0065174ee5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.962 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:41.962 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.962 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.962 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.962 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.963 DEBUG:boto:Method: GET 2023-09-29T22:25:41.963 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:41.963 DEBUG:boto:Data: 2023-09-29T22:25:41.963 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.963 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.963 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.963 DEBUG:boto:Params: {} 2023-09-29T22:25:41.963 DEBUG:boto:Token: None 2023-09-29T22:25:41.963 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/?versions 2023-09-29T22:25:41.964 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE= 2023-09-29T22:25:41.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CoCYTb65oE5MOOBd1VPvspnZ4IE=', 'Content-Length': '0'} 2023-09-29T22:25:41.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006055a145aa48d9a5-0065174ee5-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.966 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:41.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:41.967 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.967 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.967 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.967 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:41.967 DEBUG:boto:Data: 2023-09-29T22:25:41.967 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.967 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.967 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.967 DEBUG:boto:Params: {} 2023-09-29T22:25:41.967 DEBUG:boto:Token: None 2023-09-29T22:25:41.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj 2023-09-29T22:25:41.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJTBKzZxYz+Hu2owgbWsDjzKUqw= 2023-09-29T22:25:41.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJTBKzZxYz+Hu2owgbWsDjzKUqw=', 'Content-Length': '0'} 2023-09-29T22:25:41.970 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003b7de73d9fba523e-0065174ee5-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.971 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.971 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.971 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.971 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:41.971 DEBUG:boto:Data: 2023-09-29T22:25:41.972 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.972 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.972 DEBUG:boto:Params: {} 2023-09-29T22:25:41.973 DEBUG:boto:Token: None 2023-09-29T22:25:41.973 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj 2023-09-29T22:25:41.973 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJTBKzZxYz+Hu2owgbWsDjzKUqw= 2023-09-29T22:25:41.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJTBKzZxYz+Hu2owgbWsDjzKUqw=', 'Content-Length': '0'} 2023-09-29T22:25:41.975 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fed6b30d07b381f0-0065174ee5-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:41.976 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.976 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.976 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.977 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.977 DEBUG:boto:Method: GET 2023-09-29T22:25:41.977 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.977 DEBUG:boto:Data: 2023-09-29T22:25:41.977 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.977 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.978 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.978 DEBUG:boto:Params: {} 2023-09-29T22:25:41.978 DEBUG:boto:Token: None 2023-09-29T22:25:41.978 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.978 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88= 2023-09-29T22:25:41.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88=', 'Content-Length': '0'} 2023-09-29T22:25:41.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc33cfff8a6d404d-0065174ee5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.980 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.980 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.980 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.980 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.981 DEBUG:boto:Method: GET 2023-09-29T22:25:41.981 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.981 DEBUG:boto:Data: 2023-09-29T22:25:41.981 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.981 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.981 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.981 DEBUG:boto:Params: {} 2023-09-29T22:25:41.981 DEBUG:boto:Token: None 2023-09-29T22:25:41.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.981 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88= 2023-09-29T22:25:41.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88=', 'Content-Length': '0'} 2023-09-29T22:25:41.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c99597f5d203ecb9-0065174ee5-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.983 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.984 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.984 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.984 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.984 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.984 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.984 DEBUG:boto:Data: 2023-09-29T22:25:41.984 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.984 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.984 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.984 DEBUG:boto:Params: {} 2023-09-29T22:25:41.985 DEBUG:boto:Token: None 2023-09-29T22:25:41.985 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.985 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kNpCDQgvnO8nhrH/1aH4tXnJ50c= 2023-09-29T22:25:41.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNpCDQgvnO8nhrH/1aH4tXnJ50c=', 'Content-Length': '0'} 2023-09-29T22:25:41.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000920474507cd13216-0065174ee5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.987 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.987 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.987 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.987 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.988 DEBUG:boto:Method: HEAD 2023-09-29T22:25:41.988 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.988 DEBUG:boto:Data: 2023-09-29T22:25:41.988 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.988 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.988 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.988 DEBUG:boto:Params: {} 2023-09-29T22:25:41.988 DEBUG:boto:Token: None 2023-09-29T22:25:41.988 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.989 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kNpCDQgvnO8nhrH/1aH4tXnJ50c= 2023-09-29T22:25:41.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNpCDQgvnO8nhrH/1aH4tXnJ50c=', 'Content-Length': '0'} 2023-09-29T22:25:41.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082ae4592f3fc656e-0065174ee5-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:41.991 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.991 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.991 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.991 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.991 DEBUG:boto:Method: GET 2023-09-29T22:25:41.991 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.991 DEBUG:boto:Data: 2023-09-29T22:25:41.991 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.991 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.992 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.992 DEBUG:boto:Params: {} 2023-09-29T22:25:41.992 DEBUG:boto:Token: None 2023-09-29T22:25:41.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.992 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88= 2023-09-29T22:25:41.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88=', 'Content-Length': '0'} 2023-09-29T22:25:41.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000758373cbbc429dc0-0065174ee5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.994 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:41.994 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:41.994 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.994 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.994 DEBUG:boto:Method: GET 2023-09-29T22:25:41.995 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.995 DEBUG:boto:Data: 2023-09-29T22:25:41.995 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:41.995 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.995 DEBUG:boto:Params: {} 2023-09-29T22:25:41.995 DEBUG:boto:Token: None 2023-09-29T22:25:41.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:41.995 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88= 2023-09-29T22:25:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ud9ssjTshtkmMO9NQy2QpA3Oq88=', 'Content-Length': '0'} 2023-09-29T22:25:41.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000848558bf0af7b7af-0065174ee5-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:41.997 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:41.997 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:41.998 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:41.998 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:41.998 DEBUG:boto:Method: GET 2023-09-29T22:25:41.998 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:41.998 DEBUG:boto:Data: 2023-09-29T22:25:41.998 DEBUG:boto:Headers: {} 2023-09-29T22:25:41.998 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:41.998 DEBUG:boto:Port: 8000 2023-09-29T22:25:41.998 DEBUG:boto:Params: {} 2023-09-29T22:25:41.999 DEBUG:boto:Token: None 2023-09-29T22:25:41.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:41 GMT /dmuhsu-21/ 2023-09-29T22:25:41.999 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DltxerUYOWdLymV5kDUpwSnuQGk= 2023-09-29T22:25:41.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DltxerUYOWdLymV5kDUpwSnuQGk=', 'Content-Length': '0'} 2023-09-29T22:25:42.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83ed387686a45d8-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.042 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.043 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.043 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.043 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.043 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.043 DEBUG:boto:Method: GET 2023-09-29T22:25:42.043 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.044 DEBUG:boto:Data: 2023-09-29T22:25:42.044 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.044 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.044 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.044 DEBUG:boto:Params: {} 2023-09-29T22:25:42.044 DEBUG:boto:Token: None 2023-09-29T22:25:42.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0= 2023-09-29T22:25:42.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0=', 'Content-Length': '0'} 2023-09-29T22:25:42.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c4a0c7a91ccf869-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.047 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.047 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.048 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.048 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.048 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.048 DEBUG:boto:Data: 2023-09-29T22:25:42.048 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.048 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.048 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.048 DEBUG:boto:Params: {} 2023-09-29T22:25:42.048 DEBUG:boto:Token: None 2023-09-29T22:25:42.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.049 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.051 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c7f2a4cc767ba632-0065174ee6-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.051 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.051 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.051 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.051 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.051 DEBUG:boto:Data: 2023-09-29T22:25:42.051 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.052 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.052 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.052 DEBUG:boto:Params: {} 2023-09-29T22:25:42.052 DEBUG:boto:Token: None 2023-09-29T22:25:42.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.054 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e35f5bf985a4fdfb-0065174ee6-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-21 zones={test-zone2, test-zone1} 2023-09-29T22:25:42.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-21 zones={test-zone3, test-zone1} 2023-09-29T22:25:42.055 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.055 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.055 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.055 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:42.055 DEBUG:boto:Data: 2023-09-29T22:25:42.055 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.055 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.055 DEBUG:boto:Params: {} 2023-09-29T22:25:42.055 DEBUG:boto:Token: None 2023-09-29T22:25:42.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA= 2023-09-29T22:25:42.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA=', 'Content-Length': '0'} 2023-09-29T22:25:42.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004108e57259fcf241-0065174ee6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.058 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.058 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.059 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.059 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:42.059 DEBUG:boto:Data: 2023-09-29T22:25:42.059 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.059 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.059 DEBUG:boto:Params: {} 2023-09-29T22:25:42.059 DEBUG:boto:Token: None 2023-09-29T22:25:42.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA= 2023-09-29T22:25:42.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA=', 'Content-Length': '0'} 2023-09-29T22:25:42.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029f3a0734b87e891-0065174ee6-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:25:42.062 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.062 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.062 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.062 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.063 DEBUG:boto:Method: GET 2023-09-29T22:25:42.063 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.063 DEBUG:boto:Data: 2023-09-29T22:25:42.063 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.063 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.063 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.063 DEBUG:boto:Params: {} 2023-09-29T22:25:42.063 DEBUG:boto:Token: None 2023-09-29T22:25:42.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2323b59d6e74c0d-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.066 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.067 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:25:42.067 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.067 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.067 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.067 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.067 DEBUG:boto:Method: GET 2023-09-29T22:25:42.067 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.068 DEBUG:boto:Data: 2023-09-29T22:25:42.068 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.068 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.068 DEBUG:boto:Params: {} 2023-09-29T22:25:42.068 DEBUG:boto:Token: None 2023-09-29T22:25:42.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce464a1798790b8-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.071 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.071 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.072 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.072 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.072 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.072 DEBUG:boto:Method: GET 2023-09-29T22:25:42.072 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.072 DEBUG:boto:Data: 2023-09-29T22:25:42.072 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.072 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.073 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.073 DEBUG:boto:Params: {} 2023-09-29T22:25:42.073 DEBUG:boto:Token: None 2023-09-29T22:25:42.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39c536a49e3163e-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.075 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.076 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.076 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.076 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.076 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.076 DEBUG:boto:Method: GET 2023-09-29T22:25:42.076 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.076 DEBUG:boto:Data: 2023-09-29T22:25:42.076 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.077 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.077 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.077 DEBUG:boto:Params: {} 2023-09-29T22:25:42.077 DEBUG:boto:Token: None 2023-09-29T22:25:42.077 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.077 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed1b97b9a8b27a4-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.122 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.123 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.123 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.123 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.123 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.123 DEBUG:boto:Data: 2023-09-29T22:25:42.123 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.124 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.124 DEBUG:boto:Params: {} 2023-09-29T22:25:42.124 DEBUG:boto:Token: None 2023-09-29T22:25:42.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.126 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d44a9387871065e9-0065174ee6-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.126 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.126 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.127 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.127 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.127 DEBUG:boto:Data: 2023-09-29T22:25:42.127 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.127 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.127 DEBUG:boto:Params: {} 2023-09-29T22:25:42.127 DEBUG:boto:Token: None 2023-09-29T22:25:42.127 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.129 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cc9e29f9953630fe-0065174ee6-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.130 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.130 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.130 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.130 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.130 DEBUG:boto:Method: GET 2023-09-29T22:25:42.130 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.130 DEBUG:boto:Data: 2023-09-29T22:25:42.131 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.131 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.131 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.131 DEBUG:boto:Params: {} 2023-09-29T22:25:42.131 DEBUG:boto:Token: None 2023-09-29T22:25:42.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.131 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb9b63179ea36f19-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.133 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.133 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.133 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.133 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.134 DEBUG:boto:Method: GET 2023-09-29T22:25:42.134 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.134 DEBUG:boto:Data: 2023-09-29T22:25:42.134 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.134 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.134 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.134 DEBUG:boto:Params: {} 2023-09-29T22:25:42.134 DEBUG:boto:Token: None 2023-09-29T22:25:42.134 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046e2c562888fc547-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.137 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.137 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.137 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.137 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.137 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.137 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.137 DEBUG:boto:Data: 2023-09-29T22:25:42.137 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.137 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.138 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.138 DEBUG:boto:Params: {} 2023-09-29T22:25:42.138 DEBUG:boto:Token: None 2023-09-29T22:25:42.138 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY= 2023-09-29T22:25:42.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY=', 'Content-Length': '0'} 2023-09-29T22:25:42.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a95a8f8c4bcad15-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.140 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.140 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.140 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.140 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.140 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.140 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.141 DEBUG:boto:Data: 2023-09-29T22:25:42.141 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.141 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.141 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.141 DEBUG:boto:Params: {} 2023-09-29T22:25:42.141 DEBUG:boto:Token: None 2023-09-29T22:25:42.141 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY= 2023-09-29T22:25:42.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY=', 'Content-Length': '0'} 2023-09-29T22:25:42.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf2b3ce162a9cef6-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.143 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.143 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.143 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.143 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.144 DEBUG:boto:Method: GET 2023-09-29T22:25:42.144 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.144 DEBUG:boto:Data: 2023-09-29T22:25:42.144 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.144 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.144 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.144 DEBUG:boto:Params: {} 2023-09-29T22:25:42.144 DEBUG:boto:Token: None 2023-09-29T22:25:42.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.145 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040318a50e81a7535-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.146 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.147 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.147 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.147 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.147 DEBUG:boto:Method: GET 2023-09-29T22:25:42.147 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.147 DEBUG:boto:Data: 2023-09-29T22:25:42.147 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.147 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.147 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.147 DEBUG:boto:Params: {} 2023-09-29T22:25:42.148 DEBUG:boto:Token: None 2023-09-29T22:25:42.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000739fc1862c374193-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.150 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.150 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.150 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.150 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.150 DEBUG:boto:Method: GET 2023-09-29T22:25:42.150 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.150 DEBUG:boto:Data: 2023-09-29T22:25:42.150 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.150 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.150 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.151 DEBUG:boto:Params: {} 2023-09-29T22:25:42.151 DEBUG:boto:Token: None 2023-09-29T22:25:42.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0= 2023-09-29T22:25:42.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0=', 'Content-Length': '0'} 2023-09-29T22:25:42.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d23aa008c4fc1f5-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.153 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.154 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.154 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.154 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.154 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.154 DEBUG:boto:Method: GET 2023-09-29T22:25:42.154 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.154 DEBUG:boto:Data: 2023-09-29T22:25:42.154 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.154 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.155 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.155 DEBUG:boto:Params: {} 2023-09-29T22:25:42.155 DEBUG:boto:Token: None 2023-09-29T22:25:42.155 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0= 2023-09-29T22:25:42.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0=', 'Content-Length': '0'} 2023-09-29T22:25:42.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005292716dce616f17-0065174ee6-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.158 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.158 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.158 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.158 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.158 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.158 DEBUG:boto:Data: 2023-09-29T22:25:42.159 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.159 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.159 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.159 DEBUG:boto:Params: {} 2023-09-29T22:25:42.159 DEBUG:boto:Token: None 2023-09-29T22:25:42.159 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.159 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.161 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003a33ec4364e99502-0065174ee6-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.161 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.161 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.161 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.161 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.161 DEBUG:boto:Data: 2023-09-29T22:25:42.162 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.162 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:25:42.162 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.162 DEBUG:boto:Params: {} 2023-09-29T22:25:42.162 DEBUG:boto:Token: None 2023-09-29T22:25:42.162 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.162 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.164 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000036681d7a012eae5e-0065174ee6-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-21 zones={test-zone3, test-zone1} 2023-09-29T22:25:42.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-21 zones={test-zone3, test-zone2} 2023-09-29T22:25:42.164 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.165 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.165 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.165 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:42.165 DEBUG:boto:Data: 2023-09-29T22:25:42.165 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.165 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.165 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.165 DEBUG:boto:Params: {} 2023-09-29T22:25:42.165 DEBUG:boto:Token: None 2023-09-29T22:25:42.166 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.166 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA= 2023-09-29T22:25:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA=', 'Content-Length': '0'} 2023-09-29T22:25:42.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a455bd7ee0c337d1-0065174ee6-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.168 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.168 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.168 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.168 DEBUG:boto:Path: /dmuhsu-21/ 2023-09-29T22:25:42.168 DEBUG:boto:Data: 2023-09-29T22:25:42.169 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.169 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.169 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.169 DEBUG:boto:Params: {} 2023-09-29T22:25:42.169 DEBUG:boto:Token: None 2023-09-29T22:25:42.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA= 2023-09-29T22:25:42.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5x2edVE8iFd7vOvMBR0sB7rp7vA=', 'Content-Length': '0'} 2023-09-29T22:25:42.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f16c0e9094b3f10a-0065174ee6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:25:42.172 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.172 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.172 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.172 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.172 DEBUG:boto:Method: GET 2023-09-29T22:25:42.172 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.172 DEBUG:boto:Data: 2023-09-29T22:25:42.172 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.172 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.173 DEBUG:boto:Params: {} 2023-09-29T22:25:42.173 DEBUG:boto:Token: None 2023-09-29T22:25:42.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.173 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ef173537ff2d58-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.175 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.176 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:25:42.176 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.176 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.176 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.176 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.177 DEBUG:boto:Method: GET 2023-09-29T22:25:42.177 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.177 DEBUG:boto:Data: 2023-09-29T22:25:42.177 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.177 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.177 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.177 DEBUG:boto:Params: {} 2023-09-29T22:25:42.177 DEBUG:boto:Token: None 2023-09-29T22:25:42.177 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.178 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef63685711481af-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.180 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:25:42.180 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.181 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.181 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.181 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.181 DEBUG:boto:Method: GET 2023-09-29T22:25:42.181 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.181 DEBUG:boto:Data: 2023-09-29T22:25:42.181 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.181 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.181 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.181 DEBUG:boto:Params: {} 2023-09-29T22:25:42.182 DEBUG:boto:Token: None 2023-09-29T22:25:42.182 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.182 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de3cb8314fe335b-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.184 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.185 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.185 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.185 DEBUG:boto:path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.185 DEBUG:boto:auth_path=/dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.185 DEBUG:boto:Method: GET 2023-09-29T22:25:42.185 DEBUG:boto:Path: /dmuhsu-21/?versions&max-keys=999 2023-09-29T22:25:42.185 DEBUG:boto:Data: 2023-09-29T22:25:42.185 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.186 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.186 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.186 DEBUG:boto:Params: {} 2023-09-29T22:25:42.186 DEBUG:boto:Token: None 2023-09-29T22:25:42.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/?versions 2023-09-29T22:25:42.186 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE= 2023-09-29T22:25:42.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qe5uJ98dE8ulkZZzU4wDT6DpWJE=', 'Content-Length': '0'} 2023-09-29T22:25:42.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdea6c0e7b1a81c-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.189 DEBUG:boto:b'dmuhsu-21999falseobjMoR.hd6ln11K0fMmlF.jLnAjvI94vkAtrue2023-09-29T22:25:11.233Zrgw-multisite-test-userMultisite Test UserNormalobjk0YdKtxCflD1pZSTF-5g5pnAifB0t-2false2023-09-29T22:25:11.218Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:42.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.189 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.189 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.189 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.189 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.190 DEBUG:boto:Data: 2023-09-29T22:25:42.190 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.190 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.190 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.190 DEBUG:boto:Params: {} 2023-09-29T22:25:42.190 DEBUG:boto:Token: None 2023-09-29T22:25:42.190 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.190 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.192 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004dbf2590497b239d-0065174ee6-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.193 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.193 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.193 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.193 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.193 DEBUG:boto:Data: 2023-09-29T22:25:42.193 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.193 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.193 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.193 DEBUG:boto:Params: {} 2023-09-29T22:25:42.193 DEBUG:boto:Token: None 2023-09-29T22:25:42.194 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.194 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.195 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e5e455a74bd8f537-0065174ee6-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.196 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.196 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.196 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.196 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.196 DEBUG:boto:Method: GET 2023-09-29T22:25:42.196 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.196 DEBUG:boto:Data: 2023-09-29T22:25:42.196 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.197 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.197 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.197 DEBUG:boto:Params: {} 2023-09-29T22:25:42.197 DEBUG:boto:Token: None 2023-09-29T22:25:42.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024b94457eb8730c1-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.199 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.199 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.199 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.199 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.200 DEBUG:boto:Method: GET 2023-09-29T22:25:42.200 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.200 DEBUG:boto:Data: 2023-09-29T22:25:42.200 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.200 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.200 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.200 DEBUG:boto:Params: {} 2023-09-29T22:25:42.200 DEBUG:boto:Token: None 2023-09-29T22:25:42.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3c20b43e1de254b-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.202 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.202 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.203 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.203 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.203 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.203 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.203 DEBUG:boto:Data: 2023-09-29T22:25:42.203 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.203 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.203 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.203 DEBUG:boto:Params: {} 2023-09-29T22:25:42.203 DEBUG:boto:Token: None 2023-09-29T22:25:42.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY= 2023-09-29T22:25:42.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY=', 'Content-Length': '0'} 2023-09-29T22:25:42.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a48dfd2439b8216-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.206 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.206 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.206 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.206 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.206 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.206 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.206 DEBUG:boto:Data: 2023-09-29T22:25:42.207 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.207 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.207 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.207 DEBUG:boto:Params: {} 2023-09-29T22:25:42.207 DEBUG:boto:Token: None 2023-09-29T22:25:42.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.207 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY= 2023-09-29T22:25:42.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FJn90TS71iUVDT2/W4wyTkpMaTY=', 'Content-Length': '0'} 2023-09-29T22:25:42.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bed5ea3aa0cbcb5-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:25:42.209 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.209 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.209 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.209 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.209 DEBUG:boto:Method: GET 2023-09-29T22:25:42.210 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.210 DEBUG:boto:Data: 2023-09-29T22:25:42.210 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.210 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.210 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.210 DEBUG:boto:Params: {} 2023-09-29T22:25:42.210 DEBUG:boto:Token: None 2023-09-29T22:25:42.210 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.210 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000708bedb7a319593a-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.212 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.212 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.213 DEBUG:boto:path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.213 DEBUG:boto:auth_path=/dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.213 DEBUG:boto:Method: GET 2023-09-29T22:25:42.213 DEBUG:boto:Path: /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.213 DEBUG:boto:Data: 2023-09-29T22:25:42.213 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.213 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.213 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.213 DEBUG:boto:Params: {} 2023-09-29T22:25:42.213 DEBUG:boto:Token: None 2023-09-29T22:25:42.214 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj?versionId=k0YdKtxCflD1pZSTF-5g5pnAifB0t-2 2023-09-29T22:25:42.214 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM= 2023-09-29T22:25:42.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hI4iT35QK2QdO8X1BqbGaXaOBsM=', 'Content-Length': '0'} 2023-09-29T22:25:42.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'k0YdKtxCflD1pZSTF-5g5pnAifB0t-2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c21f0ef001b572f-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.217 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.218 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.218 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.218 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.218 DEBUG:boto:Method: GET 2023-09-29T22:25:42.218 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.218 DEBUG:boto:Data: 2023-09-29T22:25:42.218 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.218 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.218 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.218 DEBUG:boto:Params: {} 2023-09-29T22:25:42.219 DEBUG:boto:Token: None 2023-09-29T22:25:42.219 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.219 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0= 2023-09-29T22:25:42.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0=', 'Content-Length': '0'} 2023-09-29T22:25:42.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cbc5b941157769-0065174ee6-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.262 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.263 DEBUG:boto:path=/dmuhsu-21/ 2023-09-29T22:25:42.263 DEBUG:boto:auth_path=/dmuhsu-21/ 2023-09-29T22:25:42.263 DEBUG:boto:path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.263 DEBUG:boto:auth_path=/dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.263 DEBUG:boto:Method: GET 2023-09-29T22:25:42.263 DEBUG:boto:Path: /dmuhsu-21/?prefix=obj 2023-09-29T22:25:42.263 DEBUG:boto:Data: 2023-09-29T22:25:42.263 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.263 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.263 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.264 DEBUG:boto:Params: {} 2023-09-29T22:25:42.264 DEBUG:boto:Token: None 2023-09-29T22:25:42.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/ 2023-09-29T22:25:42.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0= 2023-09-29T22:25:42.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IWSeAX9e0Fh6dBQOleC2KCb6Lu0=', 'Content-Length': '0'} 2023-09-29T22:25:42.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d06b12a709823f8b-0065174ee6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.266 DEBUG:boto:b'dmuhsu-21obj1000false' 2023-09-29T22:25:42.266 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.267 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.267 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.267 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.267 DEBUG:boto:Data: 2023-09-29T22:25:42.267 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.267 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:25:42.267 DEBUG:boto:Port: 8001 2023-09-29T22:25:42.267 DEBUG:boto:Params: {} 2023-09-29T22:25:42.267 DEBUG:boto:Token: None 2023-09-29T22:25:42.268 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.268 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.269 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002dc2c001628cf72b-0065174ee6-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.270 DEBUG:boto:path=/dmuhsu-21/obj 2023-09-29T22:25:42.270 DEBUG:boto:auth_path=/dmuhsu-21/obj 2023-09-29T22:25:42.270 DEBUG:boto:Method: HEAD 2023-09-29T22:25:42.270 DEBUG:boto:Path: /dmuhsu-21/obj 2023-09-29T22:25:42.270 DEBUG:boto:Data: 2023-09-29T22:25:42.270 DEBUG:boto:Headers: {} 2023-09-29T22:25:42.270 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:25:42.270 DEBUG:boto:Port: 8000 2023-09-29T22:25:42.270 DEBUG:boto:Params: {} 2023-09-29T22:25:42.271 DEBUG:boto:Token: None 2023-09-29T22:25:42.271 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:42 GMT /dmuhsu-21/obj 2023-09-29T22:25:42.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro= 2023-09-29T22:25:42.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lHYD05JsH2ReLwVGHKCiIUA8Fro=', 'Content-Length': '0'} 2023-09-29T22:25:42.273 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aa145e59162d8c5f-0065174ee6-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:42.273 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-21 zones={test-zone3, test-zone2} 2023-09-29T22:25:42.273 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-22 2023-09-29T22:25:42.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.273 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:42.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:42.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:42.363 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:42.363+0000 7f6d8251e880 1 bucket sync caught up with source: 2023-09-29T22:25:42.363 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2023-09-29T22:25:42.363 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2023-09-29T22:25:42.363 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:25:42.363+0000 7f6d8251e880 0 bucket checkpoint complete 2023-09-29T22:25:42.375 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-22 2023-09-29T22:25:42.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.376 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:42.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:42.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:42.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:42.473+0000 7fd636daf880 1 bucket sync caught up with source: 2023-09-29T22:25:42.474 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2023-09-29T22:25:42.474 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2023-09-29T22:25:42.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:42.473+0000 7fd636daf880 0 bucket checkpoint complete 2023-09-29T22:25:42.487 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-22 2023-09-29T22:25:42.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.487 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:42.540 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:42.541 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:42.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:42.574+0000 7f8c10a15880 1 bucket sync caught up with source: 2023-09-29T22:25:42.575 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.87.12, , ] 2023-09-29T22:25:42.575 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.87.12, , ] 2023-09-29T22:25:42.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:42.574+0000 7f8c10a15880 0 bucket checkpoint complete 2023-09-29T22:25:42.587 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-22 2023-09-29T22:25:42.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:42.587 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:42.638 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:42.638 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:42.671 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:42.670+0000 7f51f209d880 1 waiting to reach incremental sync.. 2023-09-29T22:25:47.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:47.672+0000 7f51f209d880 1 waiting to reach incremental sync.. 2023-09-29T22:25:52.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:52.673+0000 7f51f209d880 1 waiting to reach incremental sync.. 2023-09-29T22:25:57.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:25:57.674+0000 7f51f209d880 1 waiting to reach incremental sync.. 2023-09-29T22:26:02.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:02.675+0000 7f51f209d880 1 bucket sync caught up with source: 2023-09-29T22:26:02.676 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.87.12, , ] 2023-09-29T22:26:02.676 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.87.12, , ] 2023-09-29T22:26:02.676 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:02.675+0000 7f51f209d880 0 bucket checkpoint complete 2023-09-29T22:26:02.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-22 2023-09-29T22:26:02.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:02.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:02.692 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:02.748 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.748 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:02.783+0000 7f7db5010880 1 bucket sync caught up with source: 2023-09-29T22:26:02.783 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.84.6, , ] 2023-09-29T22:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.84.6, , ] 2023-09-29T22:26:02.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:02.783+0000 7f7db5010880 0 bucket checkpoint complete 2023-09-29T22:26:02.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-22 2023-09-29T22:26:02.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:02.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:02.798 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.857 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.894 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:02.894+0000 7f4a12d29880 1 bucket sync caught up with source: 2023-09-29T22:26:02.894 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.84.6, , ] 2023-09-29T22:26:02.895 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.84.6, , ] 2023-09-29T22:26:02.895 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:02.894+0000 7f4a12d29880 0 bucket checkpoint complete 2023-09-29T22:26:02.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-22 zones={test-zone2, test-zone1} 2023-09-29T22:26:02.907 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.908 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.908 DEBUG:boto:Method: HEAD 2023-09-29T22:26:02.908 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:02.908 DEBUG:boto:Data: 2023-09-29T22:26:02.908 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.908 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:02.908 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.908 DEBUG:boto:Params: {} 2023-09-29T22:26:02.909 DEBUG:boto:Token: None 2023-09-29T22:26:02.909 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/ 2023-09-29T22:26:02.909 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q2xYJ+V9cobu48tUhzrCiuiJ43c= 2023-09-29T22:26:02.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q2xYJ+V9cobu48tUhzrCiuiJ43c=', 'Content-Length': '0'} 2023-09-29T22:26:02.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3410cd6ba62d245-0065174efa-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:02.911 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.911 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.911 DEBUG:boto:Method: HEAD 2023-09-29T22:26:02.912 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:02.912 DEBUG:boto:Data: 2023-09-29T22:26:02.912 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.912 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:02.912 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.912 DEBUG:boto:Params: {} 2023-09-29T22:26:02.912 DEBUG:boto:Token: None 2023-09-29T22:26:02.912 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/ 2023-09-29T22:26:02.912 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:q2xYJ+V9cobu48tUhzrCiuiJ43c= 2023-09-29T22:26:02.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:q2xYJ+V9cobu48tUhzrCiuiJ43c=', 'Content-Length': '0'} 2023-09-29T22:26:02.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee4a3c125e369b68-0065174efa-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:02.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:02.915 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.915 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.915 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.915 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.915 DEBUG:boto:Method: GET 2023-09-29T22:26:02.915 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.915 DEBUG:boto:Data: 2023-09-29T22:26:02.915 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.916 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:02.916 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.916 DEBUG:boto:Params: {} 2023-09-29T22:26:02.916 DEBUG:boto:Token: None 2023-09-29T22:26:02.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/?versions 2023-09-29T22:26:02.916 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s= 2023-09-29T22:26:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s=', 'Content-Length': '0'} 2023-09-29T22:26:02.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525194390a6f5e82-0065174efa-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:02.962 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:02.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:02.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:02.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:02.963 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.963 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.963 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.963 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.964 DEBUG:boto:Method: GET 2023-09-29T22:26:02.964 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.964 DEBUG:boto:Data: 2023-09-29T22:26:02.964 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.964 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:02.964 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.964 DEBUG:boto:Params: {} 2023-09-29T22:26:02.964 DEBUG:boto:Token: None 2023-09-29T22:26:02.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/?versions 2023-09-29T22:26:02.965 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s= 2023-09-29T22:26:02.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s=', 'Content-Length': '0'} 2023-09-29T22:26:02.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000302eab55cf2bc0e-0065174efa-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:02.967 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:02.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:02.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:02.967 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.967 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.968 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.968 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.968 DEBUG:boto:Method: GET 2023-09-29T22:26:02.968 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.968 DEBUG:boto:Data: 2023-09-29T22:26:02.968 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.968 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:02.968 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.968 DEBUG:boto:Params: {} 2023-09-29T22:26:02.968 DEBUG:boto:Token: None 2023-09-29T22:26:02.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/?versions 2023-09-29T22:26:02.969 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s= 2023-09-29T22:26:02.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s=', 'Content-Length': '0'} 2023-09-29T22:26:02.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095e703720b8cba41-0065174efa-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:02.971 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:02.971 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:02.971 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:02.971 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.971 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.971 DEBUG:boto:Method: GET 2023-09-29T22:26:02.972 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:02.972 DEBUG:boto:Data: 2023-09-29T22:26:02.972 DEBUG:boto:Headers: {} 2023-09-29T22:26:02.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:02.972 DEBUG:boto:Port: 8000 2023-09-29T22:26:02.972 DEBUG:boto:Params: {} 2023-09-29T22:26:02.972 DEBUG:boto:Token: None 2023-09-29T22:26:02.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:02 GMT /dmuhsu-22/?versions 2023-09-29T22:26:02.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s= 2023-09-29T22:26:02.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FXMTDVNs0wQfXIJ8FqQyq80KP3s=', 'Content-Length': '0'} 2023-09-29T22:26:02.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1fd30c543525e2-0065174efa-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.018 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.019 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.019 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.019 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.019 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.019 DEBUG:boto:Data: 2023-09-29T22:26:03.019 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.020 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.020 DEBUG:boto:Params: {} 2023-09-29T22:26:03.020 DEBUG:boto:Token: None 2023-09-29T22:26:03.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.022 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006e60326d219e13f7-0065174efb-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.022 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.022 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.023 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.023 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.023 DEBUG:boto:Data: 2023-09-29T22:26:03.023 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.023 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.023 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.023 DEBUG:boto:Params: {} 2023-09-29T22:26:03.023 DEBUG:boto:Token: None 2023-09-29T22:26:03.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.025 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e178b0986f448d58-0065174efb-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.026 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.026 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.026 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.026 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.026 DEBUG:boto:Method: GET 2023-09-29T22:26:03.026 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.026 DEBUG:boto:Data: 2023-09-29T22:26:03.027 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.027 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.027 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.027 DEBUG:boto:Params: {} 2023-09-29T22:26:03.027 DEBUG:boto:Token: None 2023-09-29T22:26:03.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.027 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063c79a027b837ad0-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.029 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.029 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.029 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.029 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.029 DEBUG:boto:Method: GET 2023-09-29T22:26:03.029 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.030 DEBUG:boto:Data: 2023-09-29T22:26:03.030 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.030 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.030 DEBUG:boto:Params: {} 2023-09-29T22:26:03.030 DEBUG:boto:Token: None 2023-09-29T22:26:03.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.030 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc669c3c979f8123-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.032 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.032 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.032 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.033 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.033 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.033 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.033 DEBUG:boto:Data: 2023-09-29T22:26:03.033 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.033 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.033 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.033 DEBUG:boto:Params: {} 2023-09-29T22:26:03.033 DEBUG:boto:Token: None 2023-09-29T22:26:03.034 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.034 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cc92ea02505b633-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.036 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.036 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.036 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.036 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.036 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.036 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.036 DEBUG:boto:Data: 2023-09-29T22:26:03.036 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.037 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.037 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.037 DEBUG:boto:Params: {} 2023-09-29T22:26:03.037 DEBUG:boto:Token: None 2023-09-29T22:26:03.037 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.037 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b5833c369fc4e4c-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.039 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.039 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.039 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.040 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.040 DEBUG:boto:Method: GET 2023-09-29T22:26:03.040 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.040 DEBUG:boto:Data: 2023-09-29T22:26:03.040 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.040 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.040 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.040 DEBUG:boto:Params: {} 2023-09-29T22:26:03.040 DEBUG:boto:Token: None 2023-09-29T22:26:03.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.041 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000280acd491a2f724-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.042 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.043 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.043 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.043 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.043 DEBUG:boto:Method: GET 2023-09-29T22:26:03.043 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.043 DEBUG:boto:Data: 2023-09-29T22:26:03.043 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.043 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.043 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.043 DEBUG:boto:Params: {} 2023-09-29T22:26:03.044 DEBUG:boto:Token: None 2023-09-29T22:26:03.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.044 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bb22fe53004a6e6-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.045 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.046 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.046 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.046 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.046 DEBUG:boto:Method: GET 2023-09-29T22:26:03.046 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.046 DEBUG:boto:Data: 2023-09-29T22:26:03.046 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.046 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.046 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.046 DEBUG:boto:Params: {} 2023-09-29T22:26:03.047 DEBUG:boto:Token: None 2023-09-29T22:26:03.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf8a5ae26ac0696-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.049 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.050 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.050 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.050 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.050 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.050 DEBUG:boto:Method: GET 2023-09-29T22:26:03.050 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.050 DEBUG:boto:Data: 2023-09-29T22:26:03.050 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.050 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.051 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.051 DEBUG:boto:Params: {} 2023-09-29T22:26:03.051 DEBUG:boto:Token: None 2023-09-29T22:26:03.051 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.051 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeaf84d24fb6dcb8-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.053 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.053 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.054 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.054 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.054 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.054 DEBUG:boto:Data: 2023-09-29T22:26:03.054 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.054 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.054 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.054 DEBUG:boto:Params: {} 2023-09-29T22:26:03.054 DEBUG:boto:Token: None 2023-09-29T22:26:03.055 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.055 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.056 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000077db85c9faeccd93-0065174efb-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.057 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.057 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.057 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.057 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.057 DEBUG:boto:Data: 2023-09-29T22:26:03.057 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.057 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.057 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.057 DEBUG:boto:Params: {} 2023-09-29T22:26:03.058 DEBUG:boto:Token: None 2023-09-29T22:26:03.058 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.060 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007fe47936e213fa53-0065174efb-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-22 zones={test-zone2, test-zone1} 2023-09-29T22:26:03.060 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-22 zones={test-zone3, test-zone1} 2023-09-29T22:26:03.060 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.060 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.060 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.060 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:03.060 DEBUG:boto:Data: 2023-09-29T22:26:03.061 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.061 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.061 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.061 DEBUG:boto:Params: {} 2023-09-29T22:26:03.061 DEBUG:boto:Token: None 2023-09-29T22:26:03.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4= 2023-09-29T22:26:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4=', 'Content-Length': '0'} 2023-09-29T22:26:03.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003268e5183f8e7996-0065174efb-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.063 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.063 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.064 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.064 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:03.064 DEBUG:boto:Data: 2023-09-29T22:26:03.064 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.064 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.064 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.064 DEBUG:boto:Params: {} 2023-09-29T22:26:03.064 DEBUG:boto:Token: None 2023-09-29T22:26:03.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4= 2023-09-29T22:26:03.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4=', 'Content-Length': '0'} 2023-09-29T22:26:03.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ac49d5f195e1baf-0065174efb-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:03.067 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.067 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.067 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.067 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.067 DEBUG:boto:Method: GET 2023-09-29T22:26:03.067 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.067 DEBUG:boto:Data: 2023-09-29T22:26:03.067 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.067 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.068 DEBUG:boto:Params: {} 2023-09-29T22:26:03.068 DEBUG:boto:Token: None 2023-09-29T22:26:03.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560f20109dc91523-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.070 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:03.071 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.071 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.071 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.071 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.071 DEBUG:boto:Method: GET 2023-09-29T22:26:03.071 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.071 DEBUG:boto:Data: 2023-09-29T22:26:03.071 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.071 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.071 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.071 DEBUG:boto:Params: {} 2023-09-29T22:26:03.072 DEBUG:boto:Token: None 2023-09-29T22:26:03.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f3b57ec4526c7bb-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.118 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.119 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.119 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.119 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.119 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.119 DEBUG:boto:Method: GET 2023-09-29T22:26:03.120 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.120 DEBUG:boto:Data: 2023-09-29T22:26:03.120 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.120 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.120 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.120 DEBUG:boto:Params: {} 2023-09-29T22:26:03.120 DEBUG:boto:Token: None 2023-09-29T22:26:03.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003230668d4678c42e-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.123 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.123 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.123 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.123 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.124 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.124 DEBUG:boto:Method: GET 2023-09-29T22:26:03.124 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.124 DEBUG:boto:Data: 2023-09-29T22:26:03.124 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.124 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.124 DEBUG:boto:Params: {} 2023-09-29T22:26:03.124 DEBUG:boto:Token: None 2023-09-29T22:26:03.125 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.125 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2468eec038bce74-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.127 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.127 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.127 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.127 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.128 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.128 DEBUG:boto:Data: 2023-09-29T22:26:03.128 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.128 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.128 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.128 DEBUG:boto:Params: {} 2023-09-29T22:26:03.128 DEBUG:boto:Token: None 2023-09-29T22:26:03.128 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.130 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a5d5d019d94deb29-0065174efb-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.130 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.130 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.130 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.130 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.130 DEBUG:boto:Data: 2023-09-29T22:26:03.131 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.131 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.131 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.131 DEBUG:boto:Params: {} 2023-09-29T22:26:03.131 DEBUG:boto:Token: None 2023-09-29T22:26:03.131 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.131 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.133 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ececda74b0448855-0065174efb-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.133 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.133 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.133 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.133 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.134 DEBUG:boto:Method: GET 2023-09-29T22:26:03.134 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.134 DEBUG:boto:Data: 2023-09-29T22:26:03.134 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.134 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.134 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.134 DEBUG:boto:Params: {} 2023-09-29T22:26:03.134 DEBUG:boto:Token: None 2023-09-29T22:26:03.134 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006565f809fd688124-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.136 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.136 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.137 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.137 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.137 DEBUG:boto:Method: GET 2023-09-29T22:26:03.137 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.137 DEBUG:boto:Data: 2023-09-29T22:26:03.137 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.137 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.137 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.137 DEBUG:boto:Params: {} 2023-09-29T22:26:03.137 DEBUG:boto:Token: None 2023-09-29T22:26:03.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cfe8806a12f466a-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.140 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.140 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.140 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.140 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.140 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.140 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.140 DEBUG:boto:Data: 2023-09-29T22:26:03.140 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.140 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.141 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.141 DEBUG:boto:Params: {} 2023-09-29T22:26:03.141 DEBUG:boto:Token: None 2023-09-29T22:26:03.141 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.141 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d8ebfe0f774f030-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.143 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.143 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.143 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.143 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.143 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.143 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.143 DEBUG:boto:Data: 2023-09-29T22:26:03.143 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.143 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.144 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.144 DEBUG:boto:Params: {} 2023-09-29T22:26:03.144 DEBUG:boto:Token: None 2023-09-29T22:26:03.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a05637d9f7832305-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.146 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.146 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.146 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.146 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.147 DEBUG:boto:Method: GET 2023-09-29T22:26:03.147 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.147 DEBUG:boto:Data: 2023-09-29T22:26:03.147 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.147 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.147 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.147 DEBUG:boto:Params: {} 2023-09-29T22:26:03.147 DEBUG:boto:Token: None 2023-09-29T22:26:03.147 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5f2e86985288687-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.149 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.149 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.149 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.150 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.150 DEBUG:boto:Method: GET 2023-09-29T22:26:03.150 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.150 DEBUG:boto:Data: 2023-09-29T22:26:03.150 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.150 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.150 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.150 DEBUG:boto:Params: {} 2023-09-29T22:26:03.150 DEBUG:boto:Token: None 2023-09-29T22:26:03.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1393d9885139a8e-0065174efb-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.153 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.153 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.153 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.153 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.153 DEBUG:boto:Method: GET 2023-09-29T22:26:03.153 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.153 DEBUG:boto:Data: 2023-09-29T22:26:03.153 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.153 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.153 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.154 DEBUG:boto:Params: {} 2023-09-29T22:26:03.154 DEBUG:boto:Token: None 2023-09-29T22:26:03.154 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.154 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f823532f4dcb2ae4-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.202 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.203 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.203 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.203 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.203 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.203 DEBUG:boto:Method: GET 2023-09-29T22:26:03.203 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.204 DEBUG:boto:Data: 2023-09-29T22:26:03.204 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.204 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.204 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.204 DEBUG:boto:Params: {} 2023-09-29T22:26:03.204 DEBUG:boto:Token: None 2023-09-29T22:26:03.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.205 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a40fcac381c466-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.207 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.207 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.207 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.207 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.208 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.208 DEBUG:boto:Data: 2023-09-29T22:26:03.208 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.208 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.208 DEBUG:boto:Params: {} 2023-09-29T22:26:03.208 DEBUG:boto:Token: None 2023-09-29T22:26:03.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.209 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.210 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c979eadd20705a08-0065174efb-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.210 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.210 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.211 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.211 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.211 DEBUG:boto:Data: 2023-09-29T22:26:03.211 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.211 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.211 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.211 DEBUG:boto:Params: {} 2023-09-29T22:26:03.211 DEBUG:boto:Token: None 2023-09-29T22:26:03.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.212 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.214 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000bd92244114cecf13-0065174efb-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-22 zones={test-zone3, test-zone1} 2023-09-29T22:26:03.214 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-22 zones={test-zone3, test-zone2} 2023-09-29T22:26:03.214 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.214 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.214 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.214 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:03.215 DEBUG:boto:Data: 2023-09-29T22:26:03.215 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.215 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.215 DEBUG:boto:Params: {} 2023-09-29T22:26:03.215 DEBUG:boto:Token: None 2023-09-29T22:26:03.215 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.215 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4= 2023-09-29T22:26:03.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4=', 'Content-Length': '0'} 2023-09-29T22:26:03.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000335db3bbcc0fe054-0065174efb-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.217 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.218 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.218 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.218 DEBUG:boto:Path: /dmuhsu-22/ 2023-09-29T22:26:03.218 DEBUG:boto:Data: 2023-09-29T22:26:03.218 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.218 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.218 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.219 DEBUG:boto:Params: {} 2023-09-29T22:26:03.219 DEBUG:boto:Token: None 2023-09-29T22:26:03.219 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.219 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4= 2023-09-29T22:26:03.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/W/mOt2cxxoy+P2PZbfkNmCD1k4=', 'Content-Length': '0'} 2023-09-29T22:26:03.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000877f05362a647a22-0065174efb-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.221 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:03.221 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.221 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.222 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.222 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.222 DEBUG:boto:Method: GET 2023-09-29T22:26:03.222 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.222 DEBUG:boto:Data: 2023-09-29T22:26:03.222 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.222 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.222 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.222 DEBUG:boto:Params: {} 2023-09-29T22:26:03.223 DEBUG:boto:Token: None 2023-09-29T22:26:03.223 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.223 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc1f4e48861f1c52-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.225 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:03.226 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.226 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.226 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.226 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.226 DEBUG:boto:Method: GET 2023-09-29T22:26:03.226 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.227 DEBUG:boto:Data: 2023-09-29T22:26:03.227 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.227 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.227 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.227 DEBUG:boto:Params: {} 2023-09-29T22:26:03.227 DEBUG:boto:Token: None 2023-09-29T22:26:03.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.227 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db2016d2c30d8db-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.230 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.230 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.230 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.231 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.231 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.231 DEBUG:boto:Method: GET 2023-09-29T22:26:03.231 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.231 DEBUG:boto:Data: 2023-09-29T22:26:03.231 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.231 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.231 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.231 DEBUG:boto:Params: {} 2023-09-29T22:26:03.231 DEBUG:boto:Token: None 2023-09-29T22:26:03.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d14158baed6448a-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.234 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.234 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.234 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.234 DEBUG:boto:path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.234 DEBUG:boto:auth_path=/dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.235 DEBUG:boto:Method: GET 2023-09-29T22:26:03.235 DEBUG:boto:Path: /dmuhsu-22/?versions&max-keys=999 2023-09-29T22:26:03.235 DEBUG:boto:Data: 2023-09-29T22:26:03.235 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.235 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.235 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.235 DEBUG:boto:Params: {} 2023-09-29T22:26:03.235 DEBUG:boto:Token: None 2023-09-29T22:26:03.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/?versions 2023-09-29T22:26:03.236 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys= 2023-09-29T22:26:03.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PaOdyUszPo+uvPZluUl9cU2Qhys=', 'Content-Length': '0'} 2023-09-29T22:26:03.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2f23f6cf0ba03c-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.238 DEBUG:boto:b'dmuhsu-22999falseobjmTCV5nmmcQfHxVdLZT90LbuEfsXE.Frtrue2023-09-29T22:25:11.259Zrgw-multisite-test-userMultisite Test UserNormalobjqnudph-S3GBYZPCkNH6iP6QC.fmNnG3false2023-09-29T22:25:11.243Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.238 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.238 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.239 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.239 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.239 DEBUG:boto:Data: 2023-09-29T22:26:03.239 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.239 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.239 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.239 DEBUG:boto:Params: {} 2023-09-29T22:26:03.239 DEBUG:boto:Token: None 2023-09-29T22:26:03.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.240 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.241 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000934e8a374570bd2f-0065174efb-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.241 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.241 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.242 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.242 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.242 DEBUG:boto:Data: 2023-09-29T22:26:03.242 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.242 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.242 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.242 DEBUG:boto:Params: {} 2023-09-29T22:26:03.242 DEBUG:boto:Token: None 2023-09-29T22:26:03.242 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.243 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.245 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006014242a14559cef-0065174efb-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.245 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.245 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.245 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.245 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.245 DEBUG:boto:Method: GET 2023-09-29T22:26:03.245 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.246 DEBUG:boto:Data: 2023-09-29T22:26:03.246 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.246 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.246 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.246 DEBUG:boto:Params: {} 2023-09-29T22:26:03.246 DEBUG:boto:Token: None 2023-09-29T22:26:03.246 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.246 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025268ca5f71e4363-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.248 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.249 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.249 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.249 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.249 DEBUG:boto:Method: GET 2023-09-29T22:26:03.249 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.249 DEBUG:boto:Data: 2023-09-29T22:26:03.249 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.249 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.249 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.249 DEBUG:boto:Params: {} 2023-09-29T22:26:03.250 DEBUG:boto:Token: None 2023-09-29T22:26:03.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.250 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072950c05643e07dd-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.252 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.252 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.252 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.252 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.252 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.252 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.253 DEBUG:boto:Data: 2023-09-29T22:26:03.253 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.253 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.253 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.253 DEBUG:boto:Params: {} 2023-09-29T22:26:03.253 DEBUG:boto:Token: None 2023-09-29T22:26:03.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.253 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf0307b2c3d3a3dc-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.255 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.255 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.255 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.255 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.255 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.256 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.256 DEBUG:boto:Data: 2023-09-29T22:26:03.256 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.256 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.256 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.256 DEBUG:boto:Params: {} 2023-09-29T22:26:03.256 DEBUG:boto:Token: None 2023-09-29T22:26:03.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU= 2023-09-29T22:26:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rH+UEzadSd99BySdU625w8Lg1IU=', 'Content-Length': '0'} 2023-09-29T22:26:03.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3851149fc52a2b8-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.259 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.259 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.259 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.259 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.259 DEBUG:boto:Method: GET 2023-09-29T22:26:03.259 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.259 DEBUG:boto:Data: 2023-09-29T22:26:03.259 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.260 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.260 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.260 DEBUG:boto:Params: {} 2023-09-29T22:26:03.260 DEBUG:boto:Token: None 2023-09-29T22:26:03.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b347bef98979a5b4-0065174efb-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.262 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.262 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.262 DEBUG:boto:path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.262 DEBUG:boto:auth_path=/dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.263 DEBUG:boto:Method: GET 2023-09-29T22:26:03.263 DEBUG:boto:Path: /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.263 DEBUG:boto:Data: 2023-09-29T22:26:03.263 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.263 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.263 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.263 DEBUG:boto:Params: {} 2023-09-29T22:26:03.263 DEBUG:boto:Token: None 2023-09-29T22:26:03.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj?versionId=qnudph-S3GBYZPCkNH6iP6QC.fmNnG3 2023-09-29T22:26:03.264 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U= 2023-09-29T22:26:03.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z922MsngrGcehr/S5S8ckmEMm4U=', 'Content-Length': '0'} 2023-09-29T22:26:03.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'qnudph-S3GBYZPCkNH6iP6QC.fmNnG3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079e1d862cd04fae1-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.265 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.265 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.266 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.266 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.266 DEBUG:boto:Method: GET 2023-09-29T22:26:03.266 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.266 DEBUG:boto:Data: 2023-09-29T22:26:03.266 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.266 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.266 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.266 DEBUG:boto:Params: {} 2023-09-29T22:26:03.266 DEBUG:boto:Token: None 2023-09-29T22:26:03.267 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009887db1f9c68dd9a-0065174efb-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.314 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.315 DEBUG:boto:path=/dmuhsu-22/ 2023-09-29T22:26:03.315 DEBUG:boto:auth_path=/dmuhsu-22/ 2023-09-29T22:26:03.315 DEBUG:boto:path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.315 DEBUG:boto:auth_path=/dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.315 DEBUG:boto:Method: GET 2023-09-29T22:26:03.315 DEBUG:boto:Path: /dmuhsu-22/?prefix=obj 2023-09-29T22:26:03.315 DEBUG:boto:Data: 2023-09-29T22:26:03.315 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.316 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.316 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.316 DEBUG:boto:Params: {} 2023-09-29T22:26:03.317 DEBUG:boto:Token: None 2023-09-29T22:26:03.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/ 2023-09-29T22:26:03.318 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg= 2023-09-29T22:26:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9t0yM7Ci6kplX0QgrGuRsokZ2Zg=', 'Content-Length': '0'} 2023-09-29T22:26:03.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050ac14bd605c509f-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.320 DEBUG:boto:b'dmuhsu-22obj1000false' 2023-09-29T22:26:03.320 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.321 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.321 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.321 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.321 DEBUG:boto:Data: 2023-09-29T22:26:03.321 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.321 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:03.321 DEBUG:boto:Port: 8001 2023-09-29T22:26:03.321 DEBUG:boto:Params: {} 2023-09-29T22:26:03.321 DEBUG:boto:Token: None 2023-09-29T22:26:03.322 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.322 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.323 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002a3b2620f99c9727-0065174efb-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.323 DEBUG:boto:path=/dmuhsu-22/obj 2023-09-29T22:26:03.324 DEBUG:boto:auth_path=/dmuhsu-22/obj 2023-09-29T22:26:03.324 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.324 DEBUG:boto:Path: /dmuhsu-22/obj 2023-09-29T22:26:03.324 DEBUG:boto:Data: 2023-09-29T22:26:03.324 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.324 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.324 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.324 DEBUG:boto:Params: {} 2023-09-29T22:26:03.325 DEBUG:boto:Token: None 2023-09-29T22:26:03.325 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-22/obj 2023-09-29T22:26:03.325 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY= 2023-09-29T22:26:03.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/+aq/JSj03oscPY9P+ovZqtt8eY=', 'Content-Length': '0'} 2023-09-29T22:26:03.327 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009f8ff495c9c7e492-0065174efb-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.327 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-22 zones={test-zone3, test-zone2} 2023-09-29T22:26:03.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-23 2023-09-29T22:26:03.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.327 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.382 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.382 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.417 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:03.417+0000 7feb5ce15880 1 bucket sync caught up with source: 2023-09-29T22:26:03.417 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.13.12, , ] 2023-09-29T22:26:03.417 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.13.12, , ] 2023-09-29T22:26:03.417 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:03.417+0000 7feb5ce15880 0 bucket checkpoint complete 2023-09-29T22:26:03.429 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-23 2023-09-29T22:26:03.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.430 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.514+0000 7ff956d36880 1 bucket sync caught up with source: 2023-09-29T22:26:03.514 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.13.12, , ] 2023-09-29T22:26:03.514 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.13.12, , ] 2023-09-29T22:26:03.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.514+0000 7ff956d36880 0 bucket checkpoint complete 2023-09-29T22:26:03.526 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-23 2023-09-29T22:26:03.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.527 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.578 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.578 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.611+0000 7f816f00f880 1 bucket sync caught up with source: 2023-09-29T22:26:03.611 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.81.12, , ] 2023-09-29T22:26:03.611 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.81.12, , ] 2023-09-29T22:26:03.611 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.611+0000 7f816f00f880 0 bucket checkpoint complete 2023-09-29T22:26:03.624 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-23 2023-09-29T22:26:03.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.624 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.676 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.676 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.711+0000 7f11f0426880 1 bucket sync caught up with source: 2023-09-29T22:26:03.711 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.81.12, , ] 2023-09-29T22:26:03.711 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.81.12, , ] 2023-09-29T22:26:03.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.711+0000 7f11f0426880 0 bucket checkpoint complete 2023-09-29T22:26:03.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-23 2023-09-29T22:26:03.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.723 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.776 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.777 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.811+0000 7f16f3e15880 1 bucket sync caught up with source: 2023-09-29T22:26:03.811 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.82.12, , ] 2023-09-29T22:26:03.811 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.82.12, , ] 2023-09-29T22:26:03.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:03.811+0000 7f16f3e15880 0 bucket checkpoint complete 2023-09-29T22:26:03.823 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-23 2023-09-29T22:26:03.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.824 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.910 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:03.909+0000 7ff2bd11e880 1 bucket sync caught up with source: 2023-09-29T22:26:03.910 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.82.12, , ] 2023-09-29T22:26:03.910 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.82.12, , ] 2023-09-29T22:26:03.910 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:03.909+0000 7ff2bd11e880 0 bucket checkpoint complete 2023-09-29T22:26:03.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-23 zones={test-zone2, test-zone1} 2023-09-29T22:26:03.922 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.923 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.923 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.923 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:03.923 DEBUG:boto:Data: 2023-09-29T22:26:03.923 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.923 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.923 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.923 DEBUG:boto:Params: {} 2023-09-29T22:26:03.923 DEBUG:boto:Token: None 2023-09-29T22:26:03.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/ 2023-09-29T22:26:03.924 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TB8x/WgDuJFZ9M/kV+6qWWG26u4= 2023-09-29T22:26:03.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TB8x/WgDuJFZ9M/kV+6qWWG26u4=', 'Content-Length': '0'} 2023-09-29T22:26:03.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034adb9bea044e5e5-0065174efb-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.926 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.926 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.926 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.926 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:03.926 DEBUG:boto:Data: 2023-09-29T22:26:03.926 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.927 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.927 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.927 DEBUG:boto:Params: {} 2023-09-29T22:26:03.929 DEBUG:boto:Token: None 2023-09-29T22:26:03.929 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/ 2023-09-29T22:26:03.929 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TB8x/WgDuJFZ9M/kV+6qWWG26u4= 2023-09-29T22:26:03.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TB8x/WgDuJFZ9M/kV+6qWWG26u4=', 'Content-Length': '0'} 2023-09-29T22:26:03.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b46ec544d71028f-0065174efb-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:03.932 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.932 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.932 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.932 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.932 DEBUG:boto:Method: GET 2023-09-29T22:26:03.933 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.933 DEBUG:boto:Data: 2023-09-29T22:26:03.933 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.933 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.933 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.933 DEBUG:boto:Params: {} 2023-09-29T22:26:03.933 DEBUG:boto:Token: None 2023-09-29T22:26:03.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/?versions 2023-09-29T22:26:03.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo= 2023-09-29T22:26:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo=', 'Content-Length': '0'} 2023-09-29T22:26:03.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733c9858d17b532c-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.936 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.936 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:03.937 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.937 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.937 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.937 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.937 DEBUG:boto:Method: GET 2023-09-29T22:26:03.937 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.937 DEBUG:boto:Data: 2023-09-29T22:26:03.937 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.937 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.937 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.938 DEBUG:boto:Params: {} 2023-09-29T22:26:03.938 DEBUG:boto:Token: None 2023-09-29T22:26:03.938 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/?versions 2023-09-29T22:26:03.938 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo= 2023-09-29T22:26:03.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo=', 'Content-Length': '0'} 2023-09-29T22:26:03.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586e2a14ef624a5f-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.941 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:03.941 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.941 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.941 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.941 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.942 DEBUG:boto:Method: GET 2023-09-29T22:26:03.942 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.942 DEBUG:boto:Data: 2023-09-29T22:26:03.942 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.942 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.942 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.942 DEBUG:boto:Params: {} 2023-09-29T22:26:03.942 DEBUG:boto:Token: None 2023-09-29T22:26:03.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/?versions 2023-09-29T22:26:03.943 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo= 2023-09-29T22:26:03.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo=', 'Content-Length': '0'} 2023-09-29T22:26:03.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6429f3a46bfbe8d-0065174efb-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.986 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.987 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:03.987 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:03.987 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.987 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.987 DEBUG:boto:Method: GET 2023-09-29T22:26:03.987 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:03.988 DEBUG:boto:Data: 2023-09-29T22:26:03.988 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.988 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.988 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.988 DEBUG:boto:Params: {} 2023-09-29T22:26:03.988 DEBUG:boto:Token: None 2023-09-29T22:26:03.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/?versions 2023-09-29T22:26:03.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo= 2023-09-29T22:26:03.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:KtENueiXwXSZcEkXCKi4HZlfOKo=', 'Content-Length': '0'} 2023-09-29T22:26:03.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a5ee8a0907bb95e-0065174efb-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.991 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:03.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.992 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:03.992 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:03.992 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.992 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:03.992 DEBUG:boto:Data: 2023-09-29T22:26:03.992 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.992 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.992 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.992 DEBUG:boto:Params: {} 2023-09-29T22:26:03.992 DEBUG:boto:Token: None 2023-09-29T22:26:03.993 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/obj 2023-09-29T22:26:03.993 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XD3u7CElzxfnbxdTY7qiq/MbBs4= 2023-09-29T22:26:03.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XD3u7CElzxfnbxdTY7qiq/MbBs4=', 'Content-Length': '0'} 2023-09-29T22:26:03.994 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000986db9ecb5de6d67-0065174efb-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.994 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:03.994 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:03.995 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.995 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:03.995 DEBUG:boto:Data: 2023-09-29T22:26:03.995 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:03.995 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.995 DEBUG:boto:Params: {} 2023-09-29T22:26:03.995 DEBUG:boto:Token: None 2023-09-29T22:26:03.995 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/obj 2023-09-29T22:26:03.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XD3u7CElzxfnbxdTY7qiq/MbBs4= 2023-09-29T22:26:03.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XD3u7CElzxfnbxdTY7qiq/MbBs4=', 'Content-Length': '0'} 2023-09-29T22:26:03.997 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f35e79f9b1bdba63-0065174efb-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:03.998 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:03.998 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:03.998 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:03.998 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:03.998 DEBUG:boto:Method: GET 2023-09-29T22:26:03.998 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:03.998 DEBUG:boto:Data: 2023-09-29T22:26:03.998 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.998 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:03.999 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.999 DEBUG:boto:Params: {} 2023-09-29T22:26:03.999 DEBUG:boto:Token: None 2023-09-29T22:26:03.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:03 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:03.999 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:chKrKKOomQQQr90CZBmiTByTFGE= 2023-09-29T22:26:03.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:chKrKKOomQQQr90CZBmiTByTFGE=', 'Content-Length': '0'} 2023-09-29T22:26:04.001 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a5aae3a056074a2-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.001 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.001 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.001 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.001 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.001 DEBUG:boto:Method: GET 2023-09-29T22:26:04.002 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.002 DEBUG:boto:Data: 2023-09-29T22:26:04.002 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.002 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.002 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.002 DEBUG:boto:Params: {} 2023-09-29T22:26:04.002 DEBUG:boto:Token: None 2023-09-29T22:26:04.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.002 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0c982dab838652f-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.004 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.004 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.004 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.005 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.005 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.005 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.005 DEBUG:boto:Data: 2023-09-29T22:26:04.005 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.005 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.005 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.005 DEBUG:boto:Params: {} 2023-09-29T22:26:04.005 DEBUG:boto:Token: None 2023-09-29T22:26:04.006 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.006 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032e649dc0e5da519-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.008 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.008 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.008 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.008 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.008 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.008 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.008 DEBUG:boto:Data: 2023-09-29T22:26:04.008 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.008 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.008 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.009 DEBUG:boto:Params: {} 2023-09-29T22:26:04.009 DEBUG:boto:Token: None 2023-09-29T22:26:04.009 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.009 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000064a3b0b45abedf80-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.011 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.011 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.011 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.012 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.012 DEBUG:boto:Method: GET 2023-09-29T22:26:04.012 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.012 DEBUG:boto:Data: 2023-09-29T22:26:04.012 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.012 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.012 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.012 DEBUG:boto:Params: {} 2023-09-29T22:26:04.013 DEBUG:boto:Token: None 2023-09-29T22:26:04.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.013 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cbd022f26c2665d-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.014 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.015 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.015 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.015 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.015 DEBUG:boto:Method: GET 2023-09-29T22:26:04.015 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.015 DEBUG:boto:Data: 2023-09-29T22:26:04.015 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.015 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.015 DEBUG:boto:Params: {} 2023-09-29T22:26:04.016 DEBUG:boto:Token: None 2023-09-29T22:26:04.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.016 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1e1f63f2c56c292-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.018 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.018 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.018 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.018 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.018 DEBUG:boto:Method: GET 2023-09-29T22:26:04.019 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.019 DEBUG:boto:Data: 2023-09-29T22:26:04.019 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.019 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.019 DEBUG:boto:Params: {} 2023-09-29T22:26:04.019 DEBUG:boto:Token: None 2023-09-29T22:26:04.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.019 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0f905e5d607646-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.066 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.067 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.067 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.067 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.067 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.067 DEBUG:boto:Method: GET 2023-09-29T22:26:04.067 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.067 DEBUG:boto:Data: 2023-09-29T22:26:04.067 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.068 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.068 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.068 DEBUG:boto:Params: {} 2023-09-29T22:26:04.068 DEBUG:boto:Token: None 2023-09-29T22:26:04.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2e97483ce39c14-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.070 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.071 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.071 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.071 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.071 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.071 DEBUG:boto:Data: 2023-09-29T22:26:04.071 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.071 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.071 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.071 DEBUG:boto:Params: {} 2023-09-29T22:26:04.072 DEBUG:boto:Token: None 2023-09-29T22:26:04.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.074 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000058392acf416fef92-0065174efc-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.074 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.074 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.074 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.074 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.074 DEBUG:boto:Data: 2023-09-29T22:26:04.074 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.074 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.075 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.075 DEBUG:boto:Params: {} 2023-09-29T22:26:04.075 DEBUG:boto:Token: None 2023-09-29T22:26:04.075 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.077 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000827d41f850f94b13-0065174efc-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-23 zones={test-zone2, test-zone1} 2023-09-29T22:26:04.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-23 zones={test-zone3, test-zone1} 2023-09-29T22:26:04.077 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.077 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.077 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.078 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:04.078 DEBUG:boto:Data: 2023-09-29T22:26:04.078 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.078 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.078 DEBUG:boto:Params: {} 2023-09-29T22:26:04.078 DEBUG:boto:Token: None 2023-09-29T22:26:04.078 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.078 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q= 2023-09-29T22:26:04.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q=', 'Content-Length': '0'} 2023-09-29T22:26:04.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007387ac4652619ea0-0065174efc-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.081 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.081 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.081 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.081 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:04.081 DEBUG:boto:Data: 2023-09-29T22:26:04.081 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.081 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.081 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.081 DEBUG:boto:Params: {} 2023-09-29T22:26:04.082 DEBUG:boto:Token: None 2023-09-29T22:26:04.082 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q= 2023-09-29T22:26:04.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q=', 'Content-Length': '0'} 2023-09-29T22:26:04.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f80b200d64c5607-0065174efc-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:04.084 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.084 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.084 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.084 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.085 DEBUG:boto:Method: GET 2023-09-29T22:26:04.085 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.085 DEBUG:boto:Data: 2023-09-29T22:26:04.085 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.085 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.085 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.085 DEBUG:boto:Params: {} 2023-09-29T22:26:04.085 DEBUG:boto:Token: None 2023-09-29T22:26:04.086 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.086 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e23ce0a88ca9757-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.088 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:04.089 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.089 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.089 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.089 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.089 DEBUG:boto:Method: GET 2023-09-29T22:26:04.089 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.089 DEBUG:boto:Data: 2023-09-29T22:26:04.089 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.089 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.090 DEBUG:boto:Params: {} 2023-09-29T22:26:04.090 DEBUG:boto:Token: None 2023-09-29T22:26:04.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2ebce86e2efcef-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.092 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.093 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.093 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.093 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.093 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.093 DEBUG:boto:Method: GET 2023-09-29T22:26:04.093 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.093 DEBUG:boto:Data: 2023-09-29T22:26:04.093 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.093 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.094 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.094 DEBUG:boto:Params: {} 2023-09-29T22:26:04.094 DEBUG:boto:Token: None 2023-09-29T22:26:04.094 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.094 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd534294d53b964-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.096 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.097 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.097 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.097 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.097 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.097 DEBUG:boto:Method: GET 2023-09-29T22:26:04.097 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.097 DEBUG:boto:Data: 2023-09-29T22:26:04.098 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.098 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.098 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.098 DEBUG:boto:Params: {} 2023-09-29T22:26:04.098 DEBUG:boto:Token: None 2023-09-29T22:26:04.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.098 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0625176ec1e8593-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.100 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.101 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.101 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.101 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.101 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.101 DEBUG:boto:Data: 2023-09-29T22:26:04.101 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.102 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.102 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.102 DEBUG:boto:Params: {} 2023-09-29T22:26:04.102 DEBUG:boto:Token: None 2023-09-29T22:26:04.102 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.102 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.104 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c70aa09cf9462109-0065174efc-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.104 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.104 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.104 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.104 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.104 DEBUG:boto:Data: 2023-09-29T22:26:04.104 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.105 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.105 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.105 DEBUG:boto:Params: {} 2023-09-29T22:26:04.105 DEBUG:boto:Token: None 2023-09-29T22:26:04.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.105 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.107 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000976b9925547adc33-0065174efc-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.107 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.107 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.107 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.107 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.107 DEBUG:boto:Method: GET 2023-09-29T22:26:04.108 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.108 DEBUG:boto:Data: 2023-09-29T22:26:04.108 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.108 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.108 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.108 DEBUG:boto:Params: {} 2023-09-29T22:26:04.108 DEBUG:boto:Token: None 2023-09-29T22:26:04.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000910203b5f280ea5-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.110 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.110 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.110 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.110 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.110 DEBUG:boto:Method: GET 2023-09-29T22:26:04.111 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.111 DEBUG:boto:Data: 2023-09-29T22:26:04.111 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.111 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.111 DEBUG:boto:Params: {} 2023-09-29T22:26:04.111 DEBUG:boto:Token: None 2023-09-29T22:26:04.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d9fd1c5b88b0336-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.113 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.113 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.113 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.113 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.113 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.114 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.114 DEBUG:boto:Data: 2023-09-29T22:26:04.114 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.114 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.114 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.114 DEBUG:boto:Params: {} 2023-09-29T22:26:04.114 DEBUG:boto:Token: None 2023-09-29T22:26:04.114 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.115 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000769c5f0ccdd8e2aa-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.116 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.116 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.116 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.116 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.116 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.117 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.117 DEBUG:boto:Data: 2023-09-29T22:26:04.117 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.117 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.117 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.117 DEBUG:boto:Params: {} 2023-09-29T22:26:04.117 DEBUG:boto:Token: None 2023-09-29T22:26:04.117 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.117 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f55bda8d6d676336-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.119 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.119 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.120 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.120 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.120 DEBUG:boto:Method: GET 2023-09-29T22:26:04.120 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.120 DEBUG:boto:Data: 2023-09-29T22:26:04.120 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.120 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.120 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.120 DEBUG:boto:Params: {} 2023-09-29T22:26:04.120 DEBUG:boto:Token: None 2023-09-29T22:26:04.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca2814a3f2294341-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.123 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.123 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.123 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.123 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.123 DEBUG:boto:Method: GET 2023-09-29T22:26:04.123 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.123 DEBUG:boto:Data: 2023-09-29T22:26:04.123 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.123 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.124 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.124 DEBUG:boto:Params: {} 2023-09-29T22:26:04.124 DEBUG:boto:Token: None 2023-09-29T22:26:04.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000514c1c47727a1d27-0065174efc-1104-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.126 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.126 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.126 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.126 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.126 DEBUG:boto:Method: GET 2023-09-29T22:26:04.126 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.126 DEBUG:boto:Data: 2023-09-29T22:26:04.127 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.127 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.127 DEBUG:boto:Params: {} 2023-09-29T22:26:04.127 DEBUG:boto:Token: None 2023-09-29T22:26:04.127 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.127 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5470f19a34ed20-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.130 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.130 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.130 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.130 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.130 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.130 DEBUG:boto:Method: GET 2023-09-29T22:26:04.130 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.131 DEBUG:boto:Data: 2023-09-29T22:26:04.131 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.131 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.131 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.131 DEBUG:boto:Params: {} 2023-09-29T22:26:04.131 DEBUG:boto:Token: None 2023-09-29T22:26:04.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.131 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ccba0566600414-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.174 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.175 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.175 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.175 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.175 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.175 DEBUG:boto:Data: 2023-09-29T22:26:04.175 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.175 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.176 DEBUG:boto:Params: {} 2023-09-29T22:26:04.176 DEBUG:boto:Token: None 2023-09-29T22:26:04.176 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.178 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000bbab0c6ac6cf259e-0065174efc-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.178 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.178 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.178 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.178 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.178 DEBUG:boto:Data: 2023-09-29T22:26:04.179 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.179 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.179 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.179 DEBUG:boto:Params: {} 2023-09-29T22:26:04.179 DEBUG:boto:Token: None 2023-09-29T22:26:04.179 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.181 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006d64819f8d63cf0e-0065174efc-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-23 zones={test-zone3, test-zone1} 2023-09-29T22:26:04.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-23 zones={test-zone3, test-zone2} 2023-09-29T22:26:04.181 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.181 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.181 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.181 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:04.182 DEBUG:boto:Data: 2023-09-29T22:26:04.182 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.182 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.182 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.182 DEBUG:boto:Params: {} 2023-09-29T22:26:04.182 DEBUG:boto:Token: None 2023-09-29T22:26:04.182 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.182 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q= 2023-09-29T22:26:04.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q=', 'Content-Length': '0'} 2023-09-29T22:26:04.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000070aabf07f120de4-0065174efc-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.184 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.184 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.184 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.185 DEBUG:boto:Path: /dmuhsu-23/ 2023-09-29T22:26:04.185 DEBUG:boto:Data: 2023-09-29T22:26:04.185 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.185 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.185 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.185 DEBUG:boto:Params: {} 2023-09-29T22:26:04.185 DEBUG:boto:Token: None 2023-09-29T22:26:04.185 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.185 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q= 2023-09-29T22:26:04.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kAetUNy3ZtSxw+LdCkOZrhS+9+Q=', 'Content-Length': '0'} 2023-09-29T22:26:04.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aaf21d5fcf0718a3-0065174efc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:04.188 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.188 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.188 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.188 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.188 DEBUG:boto:Method: GET 2023-09-29T22:26:04.188 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.188 DEBUG:boto:Data: 2023-09-29T22:26:04.188 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.188 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.189 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.189 DEBUG:boto:Params: {} 2023-09-29T22:26:04.189 DEBUG:boto:Token: None 2023-09-29T22:26:04.189 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.189 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0e144fe47a4362-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.191 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.192 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:04.192 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.192 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.192 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.192 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.192 DEBUG:boto:Method: GET 2023-09-29T22:26:04.192 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.193 DEBUG:boto:Data: 2023-09-29T22:26:04.193 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.193 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.193 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.193 DEBUG:boto:Params: {} 2023-09-29T22:26:04.193 DEBUG:boto:Token: None 2023-09-29T22:26:04.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032da47eae5e6c263-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.195 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:26:04.196 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.196 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.196 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.196 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.197 DEBUG:boto:Method: GET 2023-09-29T22:26:04.197 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.197 DEBUG:boto:Data: 2023-09-29T22:26:04.197 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.197 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.197 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.197 DEBUG:boto:Params: {} 2023-09-29T22:26:04.197 DEBUG:boto:Token: None 2023-09-29T22:26:04.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.198 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072972c6cfb705c4a-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.242 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.243 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.243 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.243 DEBUG:boto:path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.243 DEBUG:boto:auth_path=/dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.243 DEBUG:boto:Method: GET 2023-09-29T22:26:04.243 DEBUG:boto:Path: /dmuhsu-23/?versions&max-keys=999 2023-09-29T22:26:04.244 DEBUG:boto:Data: 2023-09-29T22:26:04.244 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.244 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.244 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.244 DEBUG:boto:Params: {} 2023-09-29T22:26:04.244 DEBUG:boto:Token: None 2023-09-29T22:26:04.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/?versions 2023-09-29T22:26:04.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg= 2023-09-29T22:26:04.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oGKMwLPTS+7wXJ0m2b4lrK2ACqg=', 'Content-Length': '0'} 2023-09-29T22:26:04.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829b6d3b49d51590-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.290 DEBUG:boto:b'dmuhsu-23999falseobj9vq1bAE7CzFqma6nyrYjW5tzcbokSYotrue2023-09-29T22:25:11.290Zrgw-multisite-test-userMultisite Test UserNormalobjpTWAp6pWyWaPDTejvhjZJDhE5hP24UYfalse2023-09-29T22:25:11.274Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:04.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.291 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.291 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.291 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.291 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.292 DEBUG:boto:Data: 2023-09-29T22:26:04.292 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.292 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.292 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.292 DEBUG:boto:Params: {} 2023-09-29T22:26:04.292 DEBUG:boto:Token: None 2023-09-29T22:26:04.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.294 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000021757e8ecbba264e-0065174efc-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.294 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.295 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.295 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.295 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.295 DEBUG:boto:Data: 2023-09-29T22:26:04.295 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.295 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.295 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.295 DEBUG:boto:Params: {} 2023-09-29T22:26:04.295 DEBUG:boto:Token: None 2023-09-29T22:26:04.295 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.297 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fc18acad7f82875f-0065174efc-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.298 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.298 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.298 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.298 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.298 DEBUG:boto:Method: GET 2023-09-29T22:26:04.298 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.298 DEBUG:boto:Data: 2023-09-29T22:26:04.298 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.299 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.299 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.299 DEBUG:boto:Params: {} 2023-09-29T22:26:04.299 DEBUG:boto:Token: None 2023-09-29T22:26:04.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.299 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004684bf60504ec0ad-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.301 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.301 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.301 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.301 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.302 DEBUG:boto:Method: GET 2023-09-29T22:26:04.302 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.302 DEBUG:boto:Data: 2023-09-29T22:26:04.302 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.302 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.302 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.302 DEBUG:boto:Params: {} 2023-09-29T22:26:04.302 DEBUG:boto:Token: None 2023-09-29T22:26:04.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.302 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040aa6612637e0020-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.304 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.304 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.304 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.305 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.305 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.305 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.305 DEBUG:boto:Data: 2023-09-29T22:26:04.305 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.305 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.305 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.305 DEBUG:boto:Params: {} 2023-09-29T22:26:04.305 DEBUG:boto:Token: None 2023-09-29T22:26:04.305 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.306 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f55ecfba71ceda4-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.308 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.308 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.308 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.308 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.308 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.308 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.308 DEBUG:boto:Data: 2023-09-29T22:26:04.308 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.309 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.309 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.309 DEBUG:boto:Params: {} 2023-09-29T22:26:04.309 DEBUG:boto:Token: None 2023-09-29T22:26:04.309 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.309 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic= 2023-09-29T22:26:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xPoFY/Y+4vAHWoSDPF4WOXg9Lic=', 'Content-Length': '0'} 2023-09-29T22:26:04.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ac4a298c40b56461-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:26:04.311 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.311 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.311 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.311 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.311 DEBUG:boto:Method: GET 2023-09-29T22:26:04.312 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.312 DEBUG:boto:Data: 2023-09-29T22:26:04.312 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.312 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.312 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.312 DEBUG:boto:Params: {} 2023-09-29T22:26:04.312 DEBUG:boto:Token: None 2023-09-29T22:26:04.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d7ddbda6de4c9bf-0065174efc-110a-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.314 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.314 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.314 DEBUG:boto:path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.315 DEBUG:boto:auth_path=/dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.315 DEBUG:boto:Method: GET 2023-09-29T22:26:04.315 DEBUG:boto:Path: /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.315 DEBUG:boto:Data: 2023-09-29T22:26:04.315 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.315 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.315 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.315 DEBUG:boto:Params: {} 2023-09-29T22:26:04.315 DEBUG:boto:Token: None 2023-09-29T22:26:04.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj?versionId=pTWAp6pWyWaPDTejvhjZJDhE5hP24UY 2023-09-29T22:26:04.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI= 2023-09-29T22:26:04.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SHTkLabGFRQIcFEXje99jCsmLQI=', 'Content-Length': '0'} 2023-09-29T22:26:04.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:11 GMT'), ('x-amz-version-id', 'pTWAp6pWyWaPDTejvhjZJDhE5hP24UY'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079aab8a607d81706-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.317 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.318 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.318 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.318 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.318 DEBUG:boto:Method: GET 2023-09-29T22:26:04.318 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.318 DEBUG:boto:Data: 2023-09-29T22:26:04.318 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.318 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.318 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.318 DEBUG:boto:Params: {} 2023-09-29T22:26:04.319 DEBUG:boto:Token: None 2023-09-29T22:26:04.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c0f5b66984219c-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.321 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.321 DEBUG:boto:path=/dmuhsu-23/ 2023-09-29T22:26:04.321 DEBUG:boto:auth_path=/dmuhsu-23/ 2023-09-29T22:26:04.321 DEBUG:boto:path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.322 DEBUG:boto:auth_path=/dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.322 DEBUG:boto:Method: GET 2023-09-29T22:26:04.322 DEBUG:boto:Path: /dmuhsu-23/?prefix=obj 2023-09-29T22:26:04.322 DEBUG:boto:Data: 2023-09-29T22:26:04.322 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.322 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.322 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.322 DEBUG:boto:Params: {} 2023-09-29T22:26:04.322 DEBUG:boto:Token: None 2023-09-29T22:26:04.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/ 2023-09-29T22:26:04.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE= 2023-09-29T22:26:04.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4aPfB0eDarthH38o7vNaaH1dWgE=', 'Content-Length': '0'} 2023-09-29T22:26:04.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30eaaefa3b9b2c4-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.325 DEBUG:boto:b'dmuhsu-23obj1000false' 2023-09-29T22:26:04.325 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.326 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.326 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.326 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.326 DEBUG:boto:Data: 2023-09-29T22:26:04.326 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.327 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.327 DEBUG:boto:Params: {} 2023-09-29T22:26:04.327 DEBUG:boto:Token: None 2023-09-29T22:26:04.327 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.327 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.329 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001a4335b47be94043-0065174efc-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.329 DEBUG:boto:path=/dmuhsu-23/obj 2023-09-29T22:26:04.329 DEBUG:boto:auth_path=/dmuhsu-23/obj 2023-09-29T22:26:04.330 DEBUG:boto:Method: HEAD 2023-09-29T22:26:04.330 DEBUG:boto:Path: /dmuhsu-23/obj 2023-09-29T22:26:04.330 DEBUG:boto:Data: 2023-09-29T22:26:04.330 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.330 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.330 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.330 DEBUG:boto:Params: {} 2023-09-29T22:26:04.330 DEBUG:boto:Token: None 2023-09-29T22:26:04.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-23/obj 2023-09-29T22:26:04.331 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo= 2023-09-29T22:26:04.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:o7RXf7jqo0HdK8bRpm5HWI9Hgvo=', 'Content-Length': '0'} 2023-09-29T22:26:04.332 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ff7e623186cbd0a9-0065174efc-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.332 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-23 zones={test-zone3, test-zone2} 2023-09-29T22:26:04.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-09-29T22:26:04.334 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-24 2023-09-29T22:26:04.334 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:26:04.334 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:26:04.334 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.334 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:26:04.335 DEBUG:boto:Data: 2023-09-29T22:26:04.335 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.335 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.335 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.335 DEBUG:boto:Params: {} 2023-09-29T22:26:04.335 DEBUG:boto:Token: None 2023-09-29T22:26:04.335 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-24/ 2023-09-29T22:26:04.335 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:altYwW1PVpRMug++n+N0gz6o85c= 2023-09-29T22:26:04.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:altYwW1PVpRMug++n+N0gz6o85c=', 'Content-Length': '0'} 2023-09-29T22:26:04.350 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000916dae852a04e228-0065174efc-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.350 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-25 2023-09-29T22:26:04.350 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:26:04.350 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:26:04.350 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.350 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:26:04.350 DEBUG:boto:Data: 2023-09-29T22:26:04.351 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.351 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.351 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.351 DEBUG:boto:Params: {} 2023-09-29T22:26:04.351 DEBUG:boto:Token: None 2023-09-29T22:26:04.351 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-25/ 2023-09-29T22:26:04.351 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ahYTj0fU188vPa290r/2hbY+9Wk= 2023-09-29T22:26:04.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ahYTj0fU188vPa290r/2hbY+9Wk=', 'Content-Length': '0'} 2023-09-29T22:26:04.377 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021b63e9c80b8d49d-0065174efc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.377 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-26 2023-09-29T22:26:04.377 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:26:04.377 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:26:04.377 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.377 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:26:04.377 DEBUG:boto:Data: 2023-09-29T22:26:04.378 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.378 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.378 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.378 DEBUG:boto:Params: {} 2023-09-29T22:26:04.378 DEBUG:boto:Token: None 2023-09-29T22:26:04.378 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-26/ 2023-09-29T22:26:04.378 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t2p0RqGZJsmBgphczN23xwuXAx4= 2023-09-29T22:26:04.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t2p0RqGZJsmBgphczN23xwuXAx4=', 'Content-Length': '0'} 2023-09-29T22:26:04.402 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002dbe576e9fc383a8-0065174efc-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.402 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:26:04.402 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:26:04.402 DEBUG:boto:path=/dmuhsu-24/?versioning 2023-09-29T22:26:04.402 DEBUG:boto:auth_path=/dmuhsu-24/?versioning 2023-09-29T22:26:04.403 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.403 DEBUG:boto:Path: /dmuhsu-24/?versioning 2023-09-29T22:26:04.403 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:26:04.403 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.403 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.403 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.403 DEBUG:boto:Params: {} 2023-09-29T22:26:04.403 DEBUG:boto:Token: None 2023-09-29T22:26:04.404 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-24/?versioning 2023-09-29T22:26:04.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t9xOksAna/DaIiyYkLEudUwWA/M= 2023-09-29T22:26:04.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t9xOksAna/DaIiyYkLEudUwWA/M=', 'Content-Length': '228'} 2023-09-29T22:26:04.408 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000453d976b084ad38c-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.408 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:26:04.408 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:26:04.409 DEBUG:boto:path=/dmuhsu-24/?versioning 2023-09-29T22:26:04.409 DEBUG:boto:auth_path=/dmuhsu-24/?versioning 2023-09-29T22:26:04.409 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.409 DEBUG:boto:Path: /dmuhsu-24/?versioning 2023-09-29T22:26:04.409 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:26:04.409 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.409 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:04.409 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.409 DEBUG:boto:Params: {} 2023-09-29T22:26:04.410 DEBUG:boto:Token: None 2023-09-29T22:26:04.410 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-24/?versioning 2023-09-29T22:26:04.410 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:t9xOksAna/DaIiyYkLEudUwWA/M= 2023-09-29T22:26:04.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:t9xOksAna/DaIiyYkLEudUwWA/M=', 'Content-Length': '230'} 2023-09-29T22:26:04.414 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b367d5717e3f8c85-0065174efc-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.414 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:26:04.414 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:26:04.414 DEBUG:boto:path=/dmuhsu-25/?versioning 2023-09-29T22:26:04.414 DEBUG:boto:auth_path=/dmuhsu-25/?versioning 2023-09-29T22:26:04.414 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.415 DEBUG:boto:Path: /dmuhsu-25/?versioning 2023-09-29T22:26:04.415 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:26:04.415 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.415 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.415 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.415 DEBUG:boto:Params: {} 2023-09-29T22:26:04.415 DEBUG:boto:Token: None 2023-09-29T22:26:04.415 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-25/?versioning 2023-09-29T22:26:04.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipEu9PNAzbpQDZqIh/9XNTNtgf0= 2023-09-29T22:26:04.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipEu9PNAzbpQDZqIh/9XNTNtgf0=', 'Content-Length': '228'} 2023-09-29T22:26:04.423 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0d6d542ed3d5ee3-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.423 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:26:04.423 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:26:04.423 DEBUG:boto:path=/dmuhsu-25/?versioning 2023-09-29T22:26:04.423 DEBUG:boto:auth_path=/dmuhsu-25/?versioning 2023-09-29T22:26:04.423 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.423 DEBUG:boto:Path: /dmuhsu-25/?versioning 2023-09-29T22:26:04.423 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:26:04.424 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.424 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:04.424 DEBUG:boto:Port: 8000 2023-09-29T22:26:04.424 DEBUG:boto:Params: {} 2023-09-29T22:26:04.424 DEBUG:boto:Token: None 2023-09-29T22:26:04.424 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-25/?versioning 2023-09-29T22:26:04.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ipEu9PNAzbpQDZqIh/9XNTNtgf0= 2023-09-29T22:26:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ipEu9PNAzbpQDZqIh/9XNTNtgf0=', 'Content-Length': '230'} 2023-09-29T22:26:04.431 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000012daa609562b303c-0065174efc-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.431 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:26:04.432 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:26:04.432 DEBUG:boto:path=/dmuhsu-26/?versioning 2023-09-29T22:26:04.432 DEBUG:boto:auth_path=/dmuhsu-26/?versioning 2023-09-29T22:26:04.432 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.432 DEBUG:boto:Path: /dmuhsu-26/?versioning 2023-09-29T22:26:04.432 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:26:04.432 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.433 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.433 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.433 DEBUG:boto:Params: {} 2023-09-29T22:26:04.433 DEBUG:boto:Token: None 2023-09-29T22:26:04.433 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-26/?versioning 2023-09-29T22:26:04.433 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9n8cANBKgltzvPcAKe7r93xoc1k= 2023-09-29T22:26:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9n8cANBKgltzvPcAKe7r93xoc1k=', 'Content-Length': '228'} 2023-09-29T22:26:04.440 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8bf8abfab61ca51-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.440 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:26:04.440 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:26:04.440 DEBUG:boto:path=/dmuhsu-26/?versioning 2023-09-29T22:26:04.440 DEBUG:boto:auth_path=/dmuhsu-26/?versioning 2023-09-29T22:26:04.440 DEBUG:boto:Method: PUT 2023-09-29T22:26:04.440 DEBUG:boto:Path: /dmuhsu-26/?versioning 2023-09-29T22:26:04.440 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:26:04.440 DEBUG:boto:Headers: {} 2023-09-29T22:26:04.441 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:04.441 DEBUG:boto:Port: 8001 2023-09-29T22:26:04.441 DEBUG:boto:Params: {} 2023-09-29T22:26:04.441 DEBUG:boto:Token: None 2023-09-29T22:26:04.441 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:04 GMT /dmuhsu-26/?versioning 2023-09-29T22:26:04.441 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9n8cANBKgltzvPcAKe7r93xoc1k= 2023-09-29T22:26:04.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9n8cANBKgltzvPcAKe7r93xoc1k=', 'Content-Length': '230'} 2023-09-29T22:26:04.447 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8fb3edce9cf03ff-0065174efc-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:04.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:04.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:04.448 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:04.493 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:04.493 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:04.528 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:26:04.559 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026364.446615_180.1", 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:26:04.446615Z" 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:26:04.560 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:04.561 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:26:04.562 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:04.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:26:04.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:04.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:04.562 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:04.615 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:04.615 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:04.645 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:04.645+0000 7f8b8bd28880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:04.646 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:04.646+0000 7f8b8bd28880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:04.657 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026364.406469_169.1", 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:26:04.406469Z", 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:26:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:26:04Z" 2023-09-29T22:26:04.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:26:04.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.406469_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.406469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:04Z" } 2023-09-29T22:26:04.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.406469_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.406469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:04.665 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:04.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.406469_169.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:04.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026364.446615_180.1 target=1_1696026364.406469_169.1 2023-09-29T22:26:09.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:09.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:09.668 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:09.723 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:09.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:09.759 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:09.758+0000 7f58e8b1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:09.760 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:09.760+0000 7f58e8b1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:09.782 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:09.783 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:09.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:09.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026364.346424_158.1", 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:26:04.346424Z", 2023-09-29T22:26:09.786 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:09.787 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:26:09Z" 2023-09-29T22:26:09.788 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:26:09.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.346424_158.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.346424Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:09Z" } 2023-09-29T22:26:09.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.346424_158.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.346424Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:09.790 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:09.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.346424_158.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:09.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026364.446615_180.1 target=1_1696026364.346424_158.1 2023-09-29T22:26:14.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:14.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:14.791 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:14.846 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:14.846 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:14.880 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:14.880+0000 7ff72a127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:14.880 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:14.880+0000 7ff72a127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:14.894 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:26:14.894 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:26:14.894 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:26:14.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:26:14.894 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:14.895 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:14.896 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:14.897 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026364.446615_180.1", 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:26:04.446615Z", 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:26:14.898 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:26:14.899 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:26:14Z" 2023-09-29T22:26:14.900 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:26:14.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.446615_180.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.446615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:14Z" } 2023-09-29T22:26:14.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.446615_180.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.446615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:14.901 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:14.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:14.902 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:26:14.902 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:26:14.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:14.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:14.902 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:14.956 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:14.956 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:14.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:14.991+0000 7fb71f4bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:14.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:14.992+0000 7fb71f4bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:15.004 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:26:15.004 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:26:15.004 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:26:15.004 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:26:15.004 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:15.005 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:15.006 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:26:15.007 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026364.406469_169.1", 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:26:04.406469Z", 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:15.008 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:15.009 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:26:14Z" 2023-09-29T22:26:15.010 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:26:15.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.406469_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.406469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:14Z" } 2023-09-29T22:26:15.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.406469_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.406469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:15.011 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:15.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.406469_169.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:15.012 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026364.446615_180.1 target=1_1696026364.406469_169.1 2023-09-29T22:26:20.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:20.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:20.013 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:20.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:20.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:20.095 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:20.095+0000 7f4fce612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:20.096 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:20.096+0000 7f4fce612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:20.107 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:26:20.108 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:20.109 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:20.110 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026364.349534_160.1", 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:20.111 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:26:04.349534Z", 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:20.112 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:20.113 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:26:20Z" 2023-09-29T22:26:20.114 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:26:20.114 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.349534_160.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.349534Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:20Z" } 2023-09-29T22:26:20.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.349534_160.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.349534Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:20.115 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:20.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.349534_160.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:20.115 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026364.446615_180.1 target=1_1696026364.349534_160.1 2023-09-29T22:26:25.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:25.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:25.116 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:25.172 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.202 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:25.202+0000 7ffa3e2c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:25.203 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:25.203+0000 7ffa3e2c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:25.215 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:26:25.215 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:26:25.215 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:25.216 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:25.217 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:25.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026364.446615_180.1", 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:26:04.446615Z", 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:25.219 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:26:25.220 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:26:25Z" 2023-09-29T22:26:25.221 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:26:25.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026364.446615_180.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:04.446615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:25Z" } 2023-09-29T22:26:25.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026364.446615_180.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:04.446615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:26:25.223 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:25.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026364.446615_180.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:26:25.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:26:25.223 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:26:25.223 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:26:25.223 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.224 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:26:25.224 DEBUG:boto:Data: 2023-09-29T22:26:25.224 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.224 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:25.224 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.224 DEBUG:boto:Params: {} 2023-09-29T22:26:25.224 DEBUG:boto:Token: None 2023-09-29T22:26:25.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-24/ 2023-09-29T22:26:25.225 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A1YJlKu9zHtnOZXgCbbGHG4oaiE= 2023-09-29T22:26:25.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A1YJlKu9zHtnOZXgCbbGHG4oaiE=', 'Content-Length': '0'} 2023-09-29T22:26:25.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aced2133548bb1f4-0065174f11-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.227 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:26:25.227 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:26:25.227 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.227 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:26:25.228 DEBUG:boto:Data: 2023-09-29T22:26:25.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:26:25.228 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:25.228 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.228 DEBUG:boto:Params: {} 2023-09-29T22:26:25.228 DEBUG:boto:Token: None 2023-09-29T22:26:25.228 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-24/obj 2023-09-29T22:26:25.228 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YXOU3koYCkqg+GCk9GcXo2s+wiM= 2023-09-29T22:26:25.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YXOU3koYCkqg+GCk9GcXo2s+wiM='} 2023-09-29T22:26:25.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2bd3fd4a261892a-0065174f11-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.271 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:26:25.271 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:26:25.271 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.271 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:26:25.271 DEBUG:boto:Data: 2023-09-29T22:26:25.271 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.272 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:25.272 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.272 DEBUG:boto:Params: {} 2023-09-29T22:26:25.272 DEBUG:boto:Token: None 2023-09-29T22:26:25.272 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-24/ 2023-09-29T22:26:25.272 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A1YJlKu9zHtnOZXgCbbGHG4oaiE= 2023-09-29T22:26:25.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A1YJlKu9zHtnOZXgCbbGHG4oaiE=', 'Content-Length': '0'} 2023-09-29T22:26:25.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001575d5468033118a-0065174f11-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.275 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:26:25.275 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:26:25.275 DEBUG:boto:Method: DELETE 2023-09-29T22:26:25.275 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:26:25.275 DEBUG:boto:Data: 2023-09-29T22:26:25.275 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.275 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:26:25.275 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.275 DEBUG:boto:Params: {} 2023-09-29T22:26:25.276 DEBUG:boto:Token: None 2023-09-29T22:26:25.276 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-24/obj 2023-09-29T22:26:25.276 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1UWjj5oHPCOyepgN+CmObwe0YLE= 2023-09-29T22:26:25.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1UWjj5oHPCOyepgN+CmObwe0YLE=', 'Content-Length': '0'} 2023-09-29T22:26:25.284 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000070901e7c49444131-0065174f11-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.284 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:26:25.284 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:26:25.284 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.284 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:26:25.284 DEBUG:boto:Data: 2023-09-29T22:26:25.284 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.284 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:25.285 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.285 DEBUG:boto:Params: {} 2023-09-29T22:26:25.285 DEBUG:boto:Token: None 2023-09-29T22:26:25.285 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-25/ 2023-09-29T22:26:25.285 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:89phiK5a0XgeSpOZWXN50uyGeKs= 2023-09-29T22:26:25.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:89phiK5a0XgeSpOZWXN50uyGeKs=', 'Content-Length': '0'} 2023-09-29T22:26:25.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf01cc99423ad40c-0065174f11-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.288 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:26:25.288 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:26:25.288 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.288 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:26:25.288 DEBUG:boto:Data: 2023-09-29T22:26:25.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:26:25.289 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:25.289 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.289 DEBUG:boto:Params: {} 2023-09-29T22:26:25.289 DEBUG:boto:Token: None 2023-09-29T22:26:25.289 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-25/obj 2023-09-29T22:26:25.289 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5VzFy6gsiae0TDSOUXaSxxSxGSM= 2023-09-29T22:26:25.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5VzFy6gsiae0TDSOUXaSxxSxGSM='} 2023-09-29T22:26:25.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abc884a1c3d7b893-0065174f11-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.296 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:26:25.296 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:26:25.297 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.297 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:26:25.297 DEBUG:boto:Data: 2023-09-29T22:26:25.297 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.297 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:25.297 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.297 DEBUG:boto:Params: {} 2023-09-29T22:26:25.297 DEBUG:boto:Token: None 2023-09-29T22:26:25.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-25/ 2023-09-29T22:26:25.298 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:89phiK5a0XgeSpOZWXN50uyGeKs= 2023-09-29T22:26:25.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:89phiK5a0XgeSpOZWXN50uyGeKs=', 'Content-Length': '0'} 2023-09-29T22:26:25.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e139731c808694e5-0065174f11-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.300 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:26:25.300 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:26:25.300 DEBUG:boto:Method: DELETE 2023-09-29T22:26:25.300 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:26:25.300 DEBUG:boto:Data: 2023-09-29T22:26:25.301 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.301 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:26:25.301 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.301 DEBUG:boto:Params: {} 2023-09-29T22:26:25.301 DEBUG:boto:Token: None 2023-09-29T22:26:25.301 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-25/obj 2023-09-29T22:26:25.301 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rPAP/J7YoBbbvZufICT0jcamU2k= 2023-09-29T22:26:25.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rPAP/J7YoBbbvZufICT0jcamU2k=', 'Content-Length': '0'} 2023-09-29T22:26:25.311 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000029017d769da2951c-0065174f11-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.311 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:26:25.311 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:26:25.311 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.311 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:26:25.311 DEBUG:boto:Data: 2023-09-29T22:26:25.311 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.312 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:25.312 DEBUG:boto:Port: 8001 2023-09-29T22:26:25.312 DEBUG:boto:Params: {} 2023-09-29T22:26:25.312 DEBUG:boto:Token: None 2023-09-29T22:26:25.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-26/ 2023-09-29T22:26:25.312 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dSPG/nE1FUEmVioQjpySOXetoQs= 2023-09-29T22:26:25.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dSPG/nE1FUEmVioQjpySOXetoQs=', 'Content-Length': '0'} 2023-09-29T22:26:25.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b41c51a40df94b57-0065174f11-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.315 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:26:25.315 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:26:25.315 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.315 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:26:25.315 DEBUG:boto:Data: 2023-09-29T22:26:25.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:26:25.315 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:25.315 DEBUG:boto:Port: 8001 2023-09-29T22:26:25.316 DEBUG:boto:Params: {} 2023-09-29T22:26:25.316 DEBUG:boto:Token: None 2023-09-29T22:26:25.316 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-26/obj 2023-09-29T22:26:25.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eoD9PTaOeiPwSWy3sTiZ8/7T3YI= 2023-09-29T22:26:25.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eoD9PTaOeiPwSWy3sTiZ8/7T3YI='} 2023-09-29T22:26:25.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fa1b43083f4a244-0065174f11-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.322 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:26:25.322 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:26:25.322 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.322 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:26:25.322 DEBUG:boto:Data: 2023-09-29T22:26:25.322 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.322 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:25.322 DEBUG:boto:Port: 8001 2023-09-29T22:26:25.322 DEBUG:boto:Params: {} 2023-09-29T22:26:25.323 DEBUG:boto:Token: None 2023-09-29T22:26:25.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-26/ 2023-09-29T22:26:25.323 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dSPG/nE1FUEmVioQjpySOXetoQs= 2023-09-29T22:26:25.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dSPG/nE1FUEmVioQjpySOXetoQs=', 'Content-Length': '0'} 2023-09-29T22:26:25.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000121bf5818ab69028-0065174f11-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.325 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:26:25.325 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:26:25.325 DEBUG:boto:Method: DELETE 2023-09-29T22:26:25.325 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:26:25.325 DEBUG:boto:Data: 2023-09-29T22:26:25.326 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.326 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:26:25.326 DEBUG:boto:Port: 8001 2023-09-29T22:26:25.326 DEBUG:boto:Params: {} 2023-09-29T22:26:25.326 DEBUG:boto:Token: None 2023-09-29T22:26:25.326 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:26:25 GMT /dmuhsu-26/obj 2023-09-29T22:26:25.326 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tikramzKA7G6ovWJ2MosFBVAZFM= 2023-09-29T22:26:25.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tikramzKA7G6ovWJ2MosFBVAZFM=', 'Content-Length': '0'} 2023-09-29T22:26:25.334 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000f5da1045a76012e7-0065174f11-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.334 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-24 2023-09-29T22:26:25.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:25.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:25.335 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:25.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:25.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.430 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:25.430+0000 7fceb900f880 1 waiting to reach incremental sync.. 2023-09-29T22:26:30.431 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:30.431+0000 7fceb900f880 1 waiting to reach incremental sync.. 2023-09-29T22:26:35.431 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:35.431+0000 7fceb900f880 1 waiting to reach incremental sync.. 2023-09-29T22:26:40.433 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:40.433+0000 7fceb900f880 1 bucket sync caught up with source: 2023-09-29T22:26:40.433 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.25.7, , ] 2023-09-29T22:26:40.433 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.25.7, , ] 2023-09-29T22:26:40.433 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:26:40.433+0000 7fceb900f880 0 bucket checkpoint complete 2023-09-29T22:26:40.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-24 2023-09-29T22:26:40.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:40.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:40.449 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:40.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:40.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:40.541+0000 7faaec015880 1 waiting to reach incremental sync.. 2023-09-29T22:26:45.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:45.542+0000 7faaec015880 1 bucket sync caught up with source: 2023-09-29T22:26:45.544 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.25.7, , ] 2023-09-29T22:26:45.544 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.25.7, , ] 2023-09-29T22:26:45.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:45.542+0000 7faaec015880 0 bucket checkpoint complete 2023-09-29T22:26:45.559 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-24 2023-09-29T22:26:45.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:45.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:45.560 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:45.617 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:45.617 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:45.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:45.652+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:26:50.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:50.654+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:26:55.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:26:55.654+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:00.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:00.655+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:05.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:05.656+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:10.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:10.656+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:15.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:15.657+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:20.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:20.658+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:25.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:25.658+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:30.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:30.658+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:35.659 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:35.659+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:40.660 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:40.659+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:45.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:45.660+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:50.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:50.661+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:55.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:27:55.662+0000 7f73e1321880 1 waiting to reach incremental sync.. 2023-09-29T22:27:59.917 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:27:59.917+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:28:00.663 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.663+0000 7f73e1321880 1 bucket sync caught up with source: 2023-09-29T22:28:00.664 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.117.14, , ] 2023-09-29T22:28:00.664 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.117.14, , ] 2023-09-29T22:28:00.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.663+0000 7f73e1321880 0 bucket checkpoint complete 2023-09-29T22:28:00.679 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-24 2023-09-29T22:28:00.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.680 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:00.737 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:00.737 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:00.773 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.773+0000 7fbaf4c15880 1 bucket sync caught up with source: 2023-09-29T22:28:00.773 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.117.14, , ] 2023-09-29T22:28:00.774 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.117.14, , ] 2023-09-29T22:28:00.774 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.773+0000 7fbaf4c15880 0 bucket checkpoint complete 2023-09-29T22:28:00.786 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-24 2023-09-29T22:28:00.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.787 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:00.839 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:00.839 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:00.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.871+0000 7f6027911880 1 bucket sync caught up with source: 2023-09-29T22:28:00.871 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.113.14, , ] 2023-09-29T22:28:00.871 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.113.14, , ] 2023-09-29T22:28:00.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:00.871+0000 7f6027911880 0 bucket checkpoint complete 2023-09-29T22:28:00.883 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-24 2023-09-29T22:28:00.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:00.884 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:00.944 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:00.944 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:00.982 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:00.982+0000 7f63f0933880 1 bucket sync caught up with source: 2023-09-29T22:28:00.983 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.113.14, , ] 2023-09-29T22:28:00.983 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.113.14, , ] 2023-09-29T22:28:00.983 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:00.982+0000 7f63f0933880 0 bucket checkpoint complete 2023-09-29T22:28:00.996 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-24 zones={test-zone2, test-zone1} 2023-09-29T22:28:00.997 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:00.997 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:00.997 DEBUG:boto:Method: HEAD 2023-09-29T22:28:00.997 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:00.997 DEBUG:boto:Data: 2023-09-29T22:28:00.997 DEBUG:boto:Headers: {} 2023-09-29T22:28:00.997 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:00.997 DEBUG:boto:Port: 8000 2023-09-29T22:28:00.998 DEBUG:boto:Params: {} 2023-09-29T22:28:00.998 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:28:00.998 DEBUG:boto:Token: None 2023-09-29T22:28:00.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:00 GMT /dmuhsu-24/ 2023-09-29T22:28:00.999 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zdsoE0z0XjqgjFnKJ5EjrOaCcgQ= 2023-09-29T22:28:00.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zdsoE0z0XjqgjFnKJ5EjrOaCcgQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e99a5c2b03cc672e-0065174f71-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.004 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.004 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.004 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.004 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:01.004 DEBUG:boto:Data: 2023-09-29T22:28:01.004 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.005 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.005 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.005 DEBUG:boto:Params: {} 2023-09-29T22:28:01.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:28:01.005 DEBUG:boto:Token: None 2023-09-29T22:28:01.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/ 2023-09-29T22:28:01.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg= 2023-09-29T22:28:01.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg=', 'Content-Length': '0'} 2023-09-29T22:28:01.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fedac33d1d28bfd4-0065174f71-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.009 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.009 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.009 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.010 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.010 DEBUG:boto:Method: GET 2023-09-29T22:28:01.010 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.010 DEBUG:boto:Data: 2023-09-29T22:28:01.010 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.010 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.010 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.010 DEBUG:boto:Params: {} 2023-09-29T22:28:01.010 DEBUG:boto:Token: None 2023-09-29T22:28:01.010 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099d9bf93e5fc8622-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.013 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.014 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.014 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.014 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.014 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.014 DEBUG:boto:Method: GET 2023-09-29T22:28:01.014 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.015 DEBUG:boto:Data: 2023-09-29T22:28:01.015 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.015 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.015 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.015 DEBUG:boto:Params: {} 2023-09-29T22:28:01.015 DEBUG:boto:Token: None 2023-09-29T22:28:01.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.015 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f552234c71d22762-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.018 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.018 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.018 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.019 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.019 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.019 DEBUG:boto:Method: GET 2023-09-29T22:28:01.019 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.019 DEBUG:boto:Data: 2023-09-29T22:28:01.019 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.019 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.019 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.020 DEBUG:boto:Params: {} 2023-09-29T22:28:01.020 DEBUG:boto:Token: None 2023-09-29T22:28:01.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.020 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a2303c9d928a25-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.022 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.023 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.023 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.023 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.023 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.023 DEBUG:boto:Method: GET 2023-09-29T22:28:01.023 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.023 DEBUG:boto:Data: 2023-09-29T22:28:01.023 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.024 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.024 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.024 DEBUG:boto:Params: {} 2023-09-29T22:28:01.024 DEBUG:boto:Token: None 2023-09-29T22:28:01.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.024 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e68cab86a1b644-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.026 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.027 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.027 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.027 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.027 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.027 DEBUG:boto:Data: 2023-09-29T22:28:01.027 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.028 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.028 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.028 DEBUG:boto:Params: {} 2023-09-29T22:28:01.028 DEBUG:boto:Token: None 2023-09-29T22:28:01.028 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.028 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.030 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006a0ab0909f72b048-0065174f71-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.030 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.030 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.030 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.030 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.030 DEBUG:boto:Data: 2023-09-29T22:28:01.030 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.030 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.030 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.031 DEBUG:boto:Params: {} 2023-09-29T22:28:01.031 DEBUG:boto:Token: None 2023-09-29T22:28:01.031 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.031 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.032 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fb762bf028a89917-0065174f71-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-24 zones={test-zone2, test-zone1} 2023-09-29T22:28:01.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-24 zones={test-zone3, test-zone1} 2023-09-29T22:28:01.033 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.033 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.033 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.033 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:01.033 DEBUG:boto:Data: 2023-09-29T22:28:01.034 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.034 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.034 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.034 DEBUG:boto:Params: {} 2023-09-29T22:28:01.034 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:28:01.034 DEBUG:boto:Token: None 2023-09-29T22:28:01.034 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/ 2023-09-29T22:28:01.035 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg= 2023-09-29T22:28:01.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg=', 'Content-Length': '0'} 2023-09-29T22:28:01.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028ede932131aabbb-0065174f71-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.038 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.038 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.038 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.038 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:01.038 DEBUG:boto:Data: 2023-09-29T22:28:01.038 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.039 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.039 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.039 DEBUG:boto:Params: {} 2023-09-29T22:28:01.039 DEBUG:boto:Token: None 2023-09-29T22:28:01.039 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/ 2023-09-29T22:28:01.039 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg= 2023-09-29T22:28:01.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg=', 'Content-Length': '0'} 2023-09-29T22:28:01.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ccf4e5ce9069d15-0065174f71-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.041 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.041 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.042 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.042 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.042 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.042 DEBUG:boto:Method: GET 2023-09-29T22:28:01.042 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.042 DEBUG:boto:Data: 2023-09-29T22:28:01.042 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.042 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.042 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.043 DEBUG:boto:Params: {} 2023-09-29T22:28:01.043 DEBUG:boto:Token: None 2023-09-29T22:28:01.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149f20656393a4a4-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.045 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.046 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.046 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.046 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.047 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.047 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.047 DEBUG:boto:Method: GET 2023-09-29T22:28:01.047 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.047 DEBUG:boto:Data: 2023-09-29T22:28:01.047 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.047 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.047 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.048 DEBUG:boto:Params: {} 2023-09-29T22:28:01.048 DEBUG:boto:Token: None 2023-09-29T22:28:01.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.048 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000233192068a0e30c6-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.050 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.051 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.051 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.051 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.051 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.051 DEBUG:boto:Method: GET 2023-09-29T22:28:01.051 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.051 DEBUG:boto:Data: 2023-09-29T22:28:01.051 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.051 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.052 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.052 DEBUG:boto:Params: {} 2023-09-29T22:28:01.052 DEBUG:boto:Token: None 2023-09-29T22:28:01.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.052 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e3f6879d0df549-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.054 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.055 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.055 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.055 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.055 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.055 DEBUG:boto:Method: GET 2023-09-29T22:28:01.055 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.055 DEBUG:boto:Data: 2023-09-29T22:28:01.055 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.056 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.056 DEBUG:boto:Params: {} 2023-09-29T22:28:01.056 DEBUG:boto:Token: None 2023-09-29T22:28:01.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007256aff9691d4825-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.058 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.059 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.059 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.059 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.059 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.059 DEBUG:boto:Data: 2023-09-29T22:28:01.059 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.059 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.059 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.060 DEBUG:boto:Params: {} 2023-09-29T22:28:01.060 DEBUG:boto:Token: None 2023-09-29T22:28:01.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.061 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f25d1d90350f065c-0065174f71-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.061 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.062 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.062 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.062 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.062 DEBUG:boto:Data: 2023-09-29T22:28:01.062 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.062 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.062 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.062 DEBUG:boto:Params: {} 2023-09-29T22:28:01.062 DEBUG:boto:Token: None 2023-09-29T22:28:01.063 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.063 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.064 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001ad93bf7ec7b9ae0-0065174f71-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.064 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-24 zones={test-zone3, test-zone1} 2023-09-29T22:28:01.064 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-24 zones={test-zone3, test-zone2} 2023-09-29T22:28:01.064 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.065 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.065 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.065 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:01.065 DEBUG:boto:Data: 2023-09-29T22:28:01.065 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.065 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.065 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.065 DEBUG:boto:Params: {} 2023-09-29T22:28:01.065 DEBUG:boto:Token: None 2023-09-29T22:28:01.065 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/ 2023-09-29T22:28:01.066 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg= 2023-09-29T22:28:01.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg=', 'Content-Length': '0'} 2023-09-29T22:28:01.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5b044de98347af5-0065174f71-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.068 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.068 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.068 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.068 DEBUG:boto:Path: /dmuhsu-24/ 2023-09-29T22:28:01.068 DEBUG:boto:Data: 2023-09-29T22:28:01.068 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.069 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.069 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.069 DEBUG:boto:Params: {} 2023-09-29T22:28:01.069 DEBUG:boto:Token: None 2023-09-29T22:28:01.069 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/ 2023-09-29T22:28:01.069 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg= 2023-09-29T22:28:01.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4zpU7u+C8ijzlV6TbOjUBtHIRjg=', 'Content-Length': '0'} 2023-09-29T22:28:01.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2a3a73aa1e4aef5-0065174f71-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.072 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.072 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.072 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.072 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.072 DEBUG:boto:Method: GET 2023-09-29T22:28:01.072 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.072 DEBUG:boto:Data: 2023-09-29T22:28:01.073 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.073 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.073 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.073 DEBUG:boto:Params: {} 2023-09-29T22:28:01.073 DEBUG:boto:Token: None 2023-09-29T22:28:01.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.073 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae8955833a778115-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.076 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.076 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.076 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.076 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.076 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.076 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.077 DEBUG:boto:Method: GET 2023-09-29T22:28:01.077 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.077 DEBUG:boto:Data: 2023-09-29T22:28:01.077 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.077 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.077 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.077 DEBUG:boto:Params: {} 2023-09-29T22:28:01.077 DEBUG:boto:Token: None 2023-09-29T22:28:01.077 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.077 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084fcbbc745277079-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.122 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.123 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.123 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.123 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.123 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.123 DEBUG:boto:Method: GET 2023-09-29T22:28:01.123 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.123 DEBUG:boto:Data: 2023-09-29T22:28:01.124 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.124 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.124 DEBUG:boto:Params: {} 2023-09-29T22:28:01.124 DEBUG:boto:Token: None 2023-09-29T22:28:01.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.124 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6ab00e98eef129-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.127 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.127 DEBUG:boto:path=/dmuhsu-24/ 2023-09-29T22:28:01.127 DEBUG:boto:auth_path=/dmuhsu-24/ 2023-09-29T22:28:01.127 DEBUG:boto:path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.127 DEBUG:boto:auth_path=/dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.127 DEBUG:boto:Method: GET 2023-09-29T22:28:01.127 DEBUG:boto:Path: /dmuhsu-24/?versions&max-keys=999 2023-09-29T22:28:01.128 DEBUG:boto:Data: 2023-09-29T22:28:01.128 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.128 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.128 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.128 DEBUG:boto:Params: {} 2023-09-29T22:28:01.128 DEBUG:boto:Token: None 2023-09-29T22:28:01.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/?versions 2023-09-29T22:28:01.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ= 2023-09-29T22:28:01.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:i9SRsXh6gw/DukYiURk/6cmXCWQ=', 'Content-Length': '0'} 2023-09-29T22:28:01.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2cfec64b6151add-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.131 DEBUG:boto:b'dmuhsu-24999falseobjnulltrue2023-09-29T22:26:25.278Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.131 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.131 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.131 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.131 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.132 DEBUG:boto:Data: 2023-09-29T22:28:01.132 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.132 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.132 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.132 DEBUG:boto:Params: {} 2023-09-29T22:28:01.132 DEBUG:boto:Token: None 2023-09-29T22:28:01.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.132 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.134 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006db06211f72d47f5-0065174f71-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.134 DEBUG:boto:path=/dmuhsu-24/obj 2023-09-29T22:28:01.134 DEBUG:boto:auth_path=/dmuhsu-24/obj 2023-09-29T22:28:01.134 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.134 DEBUG:boto:Path: /dmuhsu-24/obj 2023-09-29T22:28:01.135 DEBUG:boto:Data: 2023-09-29T22:28:01.135 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.135 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.135 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.135 DEBUG:boto:Params: {} 2023-09-29T22:28:01.135 DEBUG:boto:Token: None 2023-09-29T22:28:01.135 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-24/obj 2023-09-29T22:28:01.135 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk= 2023-09-29T22:28:01.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nSoz/IAql/mCg2A7FhdxkECN8Wk=', 'Content-Length': '0'} 2023-09-29T22:28:01.137 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f810b02cbefdc498-0065174f71-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-24 zones={test-zone3, test-zone2} 2023-09-29T22:28:01.137 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-25 2023-09-29T22:28:01.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.138 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.191 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.191 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.226 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:01.226+0000 7f9f47215880 1 bucket sync caught up with source: 2023-09-29T22:28:01.226 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.17.14, , ] 2023-09-29T22:28:01.227 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.17.14, , ] 2023-09-29T22:28:01.227 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:01.226+0000 7f9f47215880 0 bucket checkpoint complete 2023-09-29T22:28:01.239 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-25 2023-09-29T22:28:01.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.239 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.325+0000 7ff0d9c0a880 1 bucket sync caught up with source: 2023-09-29T22:28:01.325 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.17.14, , ] 2023-09-29T22:28:01.325 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.17.14, , ] 2023-09-29T22:28:01.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.325+0000 7ff0d9c0a880 0 bucket checkpoint complete 2023-09-29T22:28:01.336 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-25 2023-09-29T22:28:01.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.337 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.388 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.388 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.422+0000 7f6d35f32880 1 bucket sync caught up with source: 2023-09-29T22:28:01.422 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.249.7, , ] 2023-09-29T22:28:01.422 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.249.7, , ] 2023-09-29T22:28:01.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.422+0000 7f6d35f32880 0 bucket checkpoint complete 2023-09-29T22:28:01.434 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-25 2023-09-29T22:28:01.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.434 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.486 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.486 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.518+0000 7f851351f880 1 bucket sync caught up with source: 2023-09-29T22:28:01.518 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.249.7, , ] 2023-09-29T22:28:01.518 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.249.7, , ] 2023-09-29T22:28:01.518 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.518+0000 7f851351f880 0 bucket checkpoint complete 2023-09-29T22:28:01.530 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-25 2023-09-29T22:28:01.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.531 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.583 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.583 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.616+0000 7f11b54c0880 1 bucket sync caught up with source: 2023-09-29T22:28:01.616 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.63.14, , ] 2023-09-29T22:28:01.616 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.63.14, , ] 2023-09-29T22:28:01.616 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:01.616+0000 7f11b54c0880 0 bucket checkpoint complete 2023-09-29T22:28:01.628 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-25 2023-09-29T22:28:01.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.628 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:01.681 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.682 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:01.718 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:01.718+0000 7f7361121880 1 bucket sync caught up with source: 2023-09-29T22:28:01.718 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.63.14, , ] 2023-09-29T22:28:01.718 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.63.14, , ] 2023-09-29T22:28:01.718 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:01.718+0000 7f7361121880 0 bucket checkpoint complete 2023-09-29T22:28:01.732 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-25 zones={test-zone2, test-zone1} 2023-09-29T22:28:01.732 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.733 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.733 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.733 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.733 DEBUG:boto:Data: 2023-09-29T22:28:01.733 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.733 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.733 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.733 DEBUG:boto:Params: {} 2023-09-29T22:28:01.733 DEBUG:boto:Token: None 2023-09-29T22:28:01.734 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.734 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb369cf8301fa89e-0065174f71-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.736 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.736 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.736 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.736 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.736 DEBUG:boto:Data: 2023-09-29T22:28:01.736 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.737 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.737 DEBUG:boto:Params: {} 2023-09-29T22:28:01.737 DEBUG:boto:Token: None 2023-09-29T22:28:01.737 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d10f2c503213260d-0065174f71-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.739 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.740 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.740 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.740 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.740 DEBUG:boto:Method: GET 2023-09-29T22:28:01.740 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.740 DEBUG:boto:Data: 2023-09-29T22:28:01.740 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.740 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.740 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.741 DEBUG:boto:Params: {} 2023-09-29T22:28:01.741 DEBUG:boto:Token: None 2023-09-29T22:28:01.741 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.741 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031f5633f02b8a3c0-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.746 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.746 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.746 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.746 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.747 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.747 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.747 DEBUG:boto:Method: GET 2023-09-29T22:28:01.747 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.747 DEBUG:boto:Data: 2023-09-29T22:28:01.747 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.747 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.747 DEBUG:boto:Params: {} 2023-09-29T22:28:01.747 DEBUG:boto:Token: None 2023-09-29T22:28:01.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4725f492fd5f49f-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.790 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.791 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.791 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.791 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.791 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.791 DEBUG:boto:Method: GET 2023-09-29T22:28:01.791 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.791 DEBUG:boto:Data: 2023-09-29T22:28:01.792 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.792 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.792 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.792 DEBUG:boto:Params: {} 2023-09-29T22:28:01.792 DEBUG:boto:Token: None 2023-09-29T22:28:01.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a47784e0b38fcbd-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.794 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.795 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.795 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.795 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.795 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.795 DEBUG:boto:Method: GET 2023-09-29T22:28:01.795 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.796 DEBUG:boto:Data: 2023-09-29T22:28:01.796 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.796 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.796 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.796 DEBUG:boto:Params: {} 2023-09-29T22:28:01.796 DEBUG:boto:Token: None 2023-09-29T22:28:01.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.796 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034fdd1fab3d2dd35-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.798 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.799 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.799 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.799 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.799 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.799 DEBUG:boto:Data: 2023-09-29T22:28:01.799 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.799 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.800 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.800 DEBUG:boto:Params: {} 2023-09-29T22:28:01.800 DEBUG:boto:Token: None 2023-09-29T22:28:01.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.801 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000556535dc79d2c56d-0065174f71-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.802 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.802 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.802 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.802 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.802 DEBUG:boto:Data: 2023-09-29T22:28:01.802 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.802 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.802 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.802 DEBUG:boto:Params: {} 2023-09-29T22:28:01.803 DEBUG:boto:Token: None 2023-09-29T22:28:01.803 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.803 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.804 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000066b477fa89918f80-0065174f71-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.804 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-25 zones={test-zone2, test-zone1} 2023-09-29T22:28:01.805 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-25 zones={test-zone3, test-zone1} 2023-09-29T22:28:01.805 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.805 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.805 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.805 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.805 DEBUG:boto:Data: 2023-09-29T22:28:01.805 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.805 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.805 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.805 DEBUG:boto:Params: {} 2023-09-29T22:28:01.806 DEBUG:boto:Token: None 2023-09-29T22:28:01.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e399a59bea2c2d0a-0065174f71-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.808 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.808 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.808 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.808 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.808 DEBUG:boto:Data: 2023-09-29T22:28:01.808 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.809 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.809 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.809 DEBUG:boto:Params: {} 2023-09-29T22:28:01.809 DEBUG:boto:Token: None 2023-09-29T22:28:01.809 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9347073feb73c67-0065174f71-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.811 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.811 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.812 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.812 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.812 DEBUG:boto:Method: GET 2023-09-29T22:28:01.812 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.812 DEBUG:boto:Data: 2023-09-29T22:28:01.812 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.812 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.812 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.812 DEBUG:boto:Params: {} 2023-09-29T22:28:01.812 DEBUG:boto:Token: None 2023-09-29T22:28:01.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.813 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9339a20f0cc3bb2-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.815 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.815 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.815 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.816 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.816 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.816 DEBUG:boto:Method: GET 2023-09-29T22:28:01.816 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.816 DEBUG:boto:Data: 2023-09-29T22:28:01.816 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.816 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.816 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.816 DEBUG:boto:Params: {} 2023-09-29T22:28:01.817 DEBUG:boto:Token: None 2023-09-29T22:28:01.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001264d2f0e916a12e-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.862 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.863 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.863 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.863 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.863 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.863 DEBUG:boto:Method: GET 2023-09-29T22:28:01.863 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.863 DEBUG:boto:Data: 2023-09-29T22:28:01.864 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.864 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.864 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.864 DEBUG:boto:Params: {} 2023-09-29T22:28:01.864 DEBUG:boto:Token: None 2023-09-29T22:28:01.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.864 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000785ac5de3d7fd183-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.867 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.867 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.867 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.867 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.867 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.867 DEBUG:boto:Method: GET 2023-09-29T22:28:01.868 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.868 DEBUG:boto:Data: 2023-09-29T22:28:01.868 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.868 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.868 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.868 DEBUG:boto:Params: {} 2023-09-29T22:28:01.868 DEBUG:boto:Token: None 2023-09-29T22:28:01.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.868 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e0b52447a603c5c-0065174f71-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.871 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.871 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.871 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.871 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.871 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.871 DEBUG:boto:Data: 2023-09-29T22:28:01.871 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.872 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.872 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.872 DEBUG:boto:Params: {} 2023-09-29T22:28:01.872 DEBUG:boto:Token: None 2023-09-29T22:28:01.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.873 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000114fe873e50ce0f0-0065174f71-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.874 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.874 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.874 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.874 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.874 DEBUG:boto:Data: 2023-09-29T22:28:01.874 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.874 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:01.874 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.875 DEBUG:boto:Params: {} 2023-09-29T22:28:01.875 DEBUG:boto:Token: None 2023-09-29T22:28:01.875 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.875 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.877 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008974444e3a920114-0065174f71-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-25 zones={test-zone3, test-zone1} 2023-09-29T22:28:01.877 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-25 zones={test-zone3, test-zone2} 2023-09-29T22:28:01.877 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.877 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.877 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.877 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.877 DEBUG:boto:Data: 2023-09-29T22:28:01.877 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.878 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.878 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.878 DEBUG:boto:Params: {} 2023-09-29T22:28:01.878 DEBUG:boto:Token: None 2023-09-29T22:28:01.878 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.878 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4be49969ead162e-0065174f71-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.880 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.880 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.881 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.881 DEBUG:boto:Path: /dmuhsu-25/ 2023-09-29T22:28:01.881 DEBUG:boto:Data: 2023-09-29T22:28:01.881 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.881 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.881 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.881 DEBUG:boto:Params: {} 2023-09-29T22:28:01.881 DEBUG:boto:Token: None 2023-09-29T22:28:01.881 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/ 2023-09-29T22:28:01.881 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE= 2023-09-29T22:28:01.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qre9Yb15ty29j8TAwpOIHPFY7vE=', 'Content-Length': '0'} 2023-09-29T22:28:01.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176241fd8a2c2569-0065174f71-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.884 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.884 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.884 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.884 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.884 DEBUG:boto:Method: GET 2023-09-29T22:28:01.884 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.884 DEBUG:boto:Data: 2023-09-29T22:28:01.884 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.884 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.884 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.884 DEBUG:boto:Params: {} 2023-09-29T22:28:01.885 DEBUG:boto:Token: None 2023-09-29T22:28:01.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.885 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce5a17adc0611a0-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.934 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.935 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.935 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.935 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.935 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.935 DEBUG:boto:Method: GET 2023-09-29T22:28:01.935 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.936 DEBUG:boto:Data: 2023-09-29T22:28:01.936 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.936 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.936 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.936 DEBUG:boto:Params: {} 2023-09-29T22:28:01.936 DEBUG:boto:Token: None 2023-09-29T22:28:01.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5dbca474a114d2-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.938 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:01.939 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.939 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.939 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.939 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.939 DEBUG:boto:Method: GET 2023-09-29T22:28:01.939 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.939 DEBUG:boto:Data: 2023-09-29T22:28:01.940 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.940 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.940 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.940 DEBUG:boto:Params: {} 2023-09-29T22:28:01.940 DEBUG:boto:Token: None 2023-09-29T22:28:01.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9e798bbe04e732-0065174f71-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.943 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.943 DEBUG:boto:path=/dmuhsu-25/ 2023-09-29T22:28:01.943 DEBUG:boto:auth_path=/dmuhsu-25/ 2023-09-29T22:28:01.943 DEBUG:boto:path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.943 DEBUG:boto:auth_path=/dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.943 DEBUG:boto:Method: GET 2023-09-29T22:28:01.944 DEBUG:boto:Path: /dmuhsu-25/?versions&max-keys=999 2023-09-29T22:28:01.944 DEBUG:boto:Data: 2023-09-29T22:28:01.944 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.944 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.944 DEBUG:boto:Params: {} 2023-09-29T22:28:01.944 DEBUG:boto:Token: None 2023-09-29T22:28:01.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/?versions 2023-09-29T22:28:01.944 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo= 2023-09-29T22:28:01.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QhuEz4c8s1wHM2tlEW1HIgsopKo=', 'Content-Length': '0'} 2023-09-29T22:28:01.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b714b31683a310fc-0065174f71-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.947 DEBUG:boto:b'dmuhsu-25999falseobjnulltrue2023-09-29T22:26:25.304Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:01.947 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.947 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.947 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.948 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.948 DEBUG:boto:Data: 2023-09-29T22:28:01.948 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:01.948 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.948 DEBUG:boto:Params: {} 2023-09-29T22:28:01.948 DEBUG:boto:Token: None 2023-09-29T22:28:01.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.950 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000086a5eba8538f6c37-0065174f71-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.950 DEBUG:boto:path=/dmuhsu-25/obj 2023-09-29T22:28:01.950 DEBUG:boto:auth_path=/dmuhsu-25/obj 2023-09-29T22:28:01.950 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.950 DEBUG:boto:Path: /dmuhsu-25/obj 2023-09-29T22:28:01.950 DEBUG:boto:Data: 2023-09-29T22:28:01.950 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.950 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:01.951 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.951 DEBUG:boto:Params: {} 2023-09-29T22:28:01.951 DEBUG:boto:Token: None 2023-09-29T22:28:01.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /dmuhsu-25/obj 2023-09-29T22:28:01.951 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY= 2023-09-29T22:28:01.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nj+it7E1/d46zhL2XmJ2NVmLlSY=', 'Content-Length': '0'} 2023-09-29T22:28:01.952 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b8f48f709c1ad2ec-0065174f71-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-25 zones={test-zone3, test-zone2} 2023-09-29T22:28:01.953 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-26 2023-09-29T22:28:01.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:01.953 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.006 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.007 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.041 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:02.041+0000 7f3e25323880 1 bucket sync caught up with source: 2023-09-29T22:28:02.041 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.12.14, , ] 2023-09-29T22:28:02.041 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.12.14, , ] 2023-09-29T22:28:02.041 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:02.041+0000 7f3e25323880 0 bucket checkpoint complete 2023-09-29T22:28:02.053 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-26 2023-09-29T22:28:02.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.054 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.107 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.107 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.140+0000 7efcacb1e880 1 bucket sync caught up with source: 2023-09-29T22:28:02.141 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.12.14, , ] 2023-09-29T22:28:02.141 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.12.14, , ] 2023-09-29T22:28:02.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.140+0000 7efcacb1e880 0 bucket checkpoint complete 2023-09-29T22:28:02.152 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-26 2023-09-29T22:28:02.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.153 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.238+0000 7efffda44880 1 bucket sync caught up with source: 2023-09-29T22:28:02.238 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.78.14, , ] 2023-09-29T22:28:02.238 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.78.14, , ] 2023-09-29T22:28:02.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.238+0000 7efffda44880 0 bucket checkpoint complete 2023-09-29T22:28:02.250 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=dmuhsu-26 2023-09-29T22:28:02.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.251 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.336+0000 7f8cb400f880 1 bucket sync caught up with source: 2023-09-29T22:28:02.336 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.78.14, , ] 2023-09-29T22:28:02.336 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.78.14, , ] 2023-09-29T22:28:02.336 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.336+0000 7f8cb400f880 0 bucket checkpoint complete 2023-09-29T22:28:02.348 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=dmuhsu-26 2023-09-29T22:28:02.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.348 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.400 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.400 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.433+0000 7f7946336880 1 bucket sync caught up with source: 2023-09-29T22:28:02.433 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.315.7, , ] 2023-09-29T22:28:02.433 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.315.7, , ] 2023-09-29T22:28:02.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:02.433+0000 7f7946336880 0 bucket checkpoint complete 2023-09-29T22:28:02.445 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=dmuhsu-26 2023-09-29T22:28:02.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.446 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.499 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.500 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.534 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:02.534+0000 7f907b40f880 1 bucket sync caught up with source: 2023-09-29T22:28:02.535 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.315.7, , ] 2023-09-29T22:28:02.535 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.315.7, , ] 2023-09-29T22:28:02.535 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:02.534+0000 7f907b40f880 0 bucket checkpoint complete 2023-09-29T22:28:02.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-26 zones={test-zone2, test-zone1} 2023-09-29T22:28:02.547 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.548 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.548 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.548 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.548 DEBUG:boto:Data: 2023-09-29T22:28:02.548 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.548 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.548 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.548 DEBUG:boto:Params: {} 2023-09-29T22:28:02.549 DEBUG:boto:Token: None 2023-09-29T22:28:02.549 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.549 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab1e98f59f20942d-0065174f72-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.551 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.551 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.551 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.552 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.552 DEBUG:boto:Data: 2023-09-29T22:28:02.552 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.552 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.552 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.552 DEBUG:boto:Params: {} 2023-09-29T22:28:02.552 DEBUG:boto:Token: None 2023-09-29T22:28:02.553 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.553 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fea0162185b475d-0065174f72-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:02.555 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.555 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.555 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.556 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.556 DEBUG:boto:Method: GET 2023-09-29T22:28:02.556 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.556 DEBUG:boto:Data: 2023-09-29T22:28:02.556 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.556 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.556 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.556 DEBUG:boto:Params: {} 2023-09-29T22:28:02.557 DEBUG:boto:Token: None 2023-09-29T22:28:02.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.557 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434c1e93e7a79cc2-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.602 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:02.603 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.603 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.603 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.603 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.603 DEBUG:boto:Method: GET 2023-09-29T22:28:02.604 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.604 DEBUG:boto:Data: 2023-09-29T22:28:02.604 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.604 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.604 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.604 DEBUG:boto:Params: {} 2023-09-29T22:28:02.604 DEBUG:boto:Token: None 2023-09-29T22:28:02.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.605 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003f88d1fd6102f53-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.607 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.607 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.607 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.607 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.607 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.608 DEBUG:boto:Method: GET 2023-09-29T22:28:02.608 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.608 DEBUG:boto:Data: 2023-09-29T22:28:02.608 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.608 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.608 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.608 DEBUG:boto:Params: {} 2023-09-29T22:28:02.608 DEBUG:boto:Token: None 2023-09-29T22:28:02.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.609 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c6b71f7327787e-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.611 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.611 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.611 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.611 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.611 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.612 DEBUG:boto:Method: GET 2023-09-29T22:28:02.612 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.612 DEBUG:boto:Data: 2023-09-29T22:28:02.612 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.612 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.612 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.612 DEBUG:boto:Params: {} 2023-09-29T22:28:02.612 DEBUG:boto:Token: None 2023-09-29T22:28:02.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.613 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e8cff059b6f1b9-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.662 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:02.663 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.663 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.663 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.663 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.663 DEBUG:boto:Data: 2023-09-29T22:28:02.663 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.663 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.664 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.664 DEBUG:boto:Params: {} 2023-09-29T22:28:02.664 DEBUG:boto:Token: None 2023-09-29T22:28:02.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.664 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.666 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cc03f7e40102b5a9-0065174f72-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.666 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.666 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.666 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.666 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.666 DEBUG:boto:Data: 2023-09-29T22:28:02.667 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.667 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.667 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.667 DEBUG:boto:Params: {} 2023-09-29T22:28:02.667 DEBUG:boto:Token: None 2023-09-29T22:28:02.667 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.667 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.669 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000eb74e580613edc4c-0065174f72-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-26 zones={test-zone2, test-zone1} 2023-09-29T22:28:02.669 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-26 zones={test-zone3, test-zone1} 2023-09-29T22:28:02.669 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.669 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.669 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.669 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.669 DEBUG:boto:Data: 2023-09-29T22:28:02.669 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.670 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.670 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.670 DEBUG:boto:Params: {} 2023-09-29T22:28:02.670 DEBUG:boto:Token: None 2023-09-29T22:28:02.670 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.670 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080d2027233f75aa1-0065174f72-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.673 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.673 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.673 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.673 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.674 DEBUG:boto:Data: 2023-09-29T22:28:02.674 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.674 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.674 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.674 DEBUG:boto:Params: {} 2023-09-29T22:28:02.674 DEBUG:boto:Token: None 2023-09-29T22:28:02.674 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.674 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3c4b7b860e3ee8a-0065174f72-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.676 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:02.677 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.677 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.677 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.677 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.677 DEBUG:boto:Method: GET 2023-09-29T22:28:02.677 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.677 DEBUG:boto:Data: 2023-09-29T22:28:02.677 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.677 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.678 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.678 DEBUG:boto:Params: {} 2023-09-29T22:28:02.678 DEBUG:boto:Token: None 2023-09-29T22:28:02.678 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.678 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abaff4682af09b5b-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.680 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.681 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:02.681 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.681 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.681 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.681 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.681 DEBUG:boto:Method: GET 2023-09-29T22:28:02.681 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.681 DEBUG:boto:Data: 2023-09-29T22:28:02.681 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.682 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.682 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.682 DEBUG:boto:Params: {} 2023-09-29T22:28:02.682 DEBUG:boto:Token: None 2023-09-29T22:28:02.682 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.682 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f380800924cf017-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.685 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.685 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.685 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.685 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.685 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.685 DEBUG:boto:Method: GET 2023-09-29T22:28:02.686 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.686 DEBUG:boto:Data: 2023-09-29T22:28:02.686 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.686 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.686 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.686 DEBUG:boto:Params: {} 2023-09-29T22:28:02.686 DEBUG:boto:Token: None 2023-09-29T22:28:02.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005bfaf2ac00e39d-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.689 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.689 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.689 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.689 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.690 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.690 DEBUG:boto:Method: GET 2023-09-29T22:28:02.690 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.690 DEBUG:boto:Data: 2023-09-29T22:28:02.690 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.690 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.690 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.690 DEBUG:boto:Params: {} 2023-09-29T22:28:02.690 DEBUG:boto:Token: None 2023-09-29T22:28:02.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c39eff6ee96fd3-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.693 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:02.693 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.693 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.694 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.694 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.694 DEBUG:boto:Data: 2023-09-29T22:28:02.694 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.694 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.694 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.694 DEBUG:boto:Params: {} 2023-09-29T22:28:02.694 DEBUG:boto:Token: None 2023-09-29T22:28:02.694 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.694 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.696 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c84bb7e0a4fe6655-0065174f72-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.696 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.696 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.696 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.696 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.696 DEBUG:boto:Data: 2023-09-29T22:28:02.696 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.697 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.697 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.697 DEBUG:boto:Params: {} 2023-09-29T22:28:02.697 DEBUG:boto:Token: None 2023-09-29T22:28:02.697 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.697 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.699 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002bf54224b5be7c6f-0065174f72-1104-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-26 zones={test-zone3, test-zone1} 2023-09-29T22:28:02.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dmuhsu-26 zones={test-zone3, test-zone2} 2023-09-29T22:28:02.699 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.699 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.699 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.699 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.699 DEBUG:boto:Data: 2023-09-29T22:28:02.699 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.700 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.700 DEBUG:boto:Params: {} 2023-09-29T22:28:02.700 DEBUG:boto:Token: None 2023-09-29T22:28:02.700 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e5c492e20206425-0065174f72-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.703 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.703 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.703 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.703 DEBUG:boto:Path: /dmuhsu-26/ 2023-09-29T22:28:02.703 DEBUG:boto:Data: 2023-09-29T22:28:02.703 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.703 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.703 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.703 DEBUG:boto:Params: {} 2023-09-29T22:28:02.703 DEBUG:boto:Token: None 2023-09-29T22:28:02.704 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/ 2023-09-29T22:28:02.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM= 2023-09-29T22:28:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ms6eqS3ojQySB9Xh0aqlOJCWqLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066c7f27306de9fe9-0065174f72-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:02.706 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.706 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.706 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.706 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.706 DEBUG:boto:Method: GET 2023-09-29T22:28:02.706 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.707 DEBUG:boto:Data: 2023-09-29T22:28:02.707 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.707 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.707 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.707 DEBUG:boto:Params: {} 2023-09-29T22:28:02.707 DEBUG:boto:Token: None 2023-09-29T22:28:02.707 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.707 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d06da5285b50b6c-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.750 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:02.751 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.751 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.751 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.751 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.751 DEBUG:boto:Method: GET 2023-09-29T22:28:02.751 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.752 DEBUG:boto:Data: 2023-09-29T22:28:02.752 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.752 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.752 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.752 DEBUG:boto:Params: {} 2023-09-29T22:28:02.752 DEBUG:boto:Token: None 2023-09-29T22:28:02.752 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.752 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024fb2c223830d10e-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.755 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:28:02.755 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.755 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.755 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.755 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.756 DEBUG:boto:Method: GET 2023-09-29T22:28:02.756 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.756 DEBUG:boto:Data: 2023-09-29T22:28:02.756 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.756 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.756 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.756 DEBUG:boto:Params: {} 2023-09-29T22:28:02.756 DEBUG:boto:Token: None 2023-09-29T22:28:02.756 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.757 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b662505b078c9392-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.759 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.759 DEBUG:boto:path=/dmuhsu-26/ 2023-09-29T22:28:02.759 DEBUG:boto:auth_path=/dmuhsu-26/ 2023-09-29T22:28:02.759 DEBUG:boto:path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.759 DEBUG:boto:auth_path=/dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.760 DEBUG:boto:Method: GET 2023-09-29T22:28:02.760 DEBUG:boto:Path: /dmuhsu-26/?versions&max-keys=999 2023-09-29T22:28:02.760 DEBUG:boto:Data: 2023-09-29T22:28:02.760 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.760 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.760 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.760 DEBUG:boto:Params: {} 2023-09-29T22:28:02.760 DEBUG:boto:Token: None 2023-09-29T22:28:02.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/?versions 2023-09-29T22:28:02.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww= 2023-09-29T22:28:02.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1uPlSx4cgklvvT/GMTiiDly2lww=', 'Content-Length': '0'} 2023-09-29T22:28:02.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8967c9d3d481d17-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.763 DEBUG:boto:b'dmuhsu-26999falseobjnulltrue2023-09-29T22:26:25.329Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:02.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:28:02.763 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.763 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.763 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.763 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.763 DEBUG:boto:Data: 2023-09-29T22:28:02.764 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.764 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.764 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.764 DEBUG:boto:Params: {} 2023-09-29T22:28:02.764 DEBUG:boto:Token: None 2023-09-29T22:28:02.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.764 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.766 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001908070c57edca70-0065174f72-110a-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.766 DEBUG:boto:path=/dmuhsu-26/obj 2023-09-29T22:28:02.766 DEBUG:boto:auth_path=/dmuhsu-26/obj 2023-09-29T22:28:02.766 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.766 DEBUG:boto:Path: /dmuhsu-26/obj 2023-09-29T22:28:02.766 DEBUG:boto:Data: 2023-09-29T22:28:02.767 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.767 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.767 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.767 DEBUG:boto:Params: {} 2023-09-29T22:28:02.767 DEBUG:boto:Token: None 2023-09-29T22:28:02.767 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-26/obj 2023-09-29T22:28:02.767 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo= 2023-09-29T22:28:02.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Lrqud2jHGqD1TFjyqeYNUwHeSPo=', 'Content-Length': '0'} 2023-09-29T22:28:02.769 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000031269f6c78bf5093-0065174f72-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dmuhsu-26 zones={test-zone3, test-zone2} 2023-09-29T22:28:02.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-09-29T22:28:02.770 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-27 2023-09-29T22:28:02.770 DEBUG:boto:path=/dmuhsu-27/ 2023-09-29T22:28:02.770 DEBUG:boto:auth_path=/dmuhsu-27/ 2023-09-29T22:28:02.770 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.771 DEBUG:boto:Path: /dmuhsu-27/ 2023-09-29T22:28:02.771 DEBUG:boto:Data: 2023-09-29T22:28:02.771 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.771 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.771 DEBUG:boto:Params: {} 2023-09-29T22:28:02.771 DEBUG:boto:Token: None 2023-09-29T22:28:02.771 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-27/ 2023-09-29T22:28:02.771 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UKvw2F98oLYcDpsoyBEdtCIiO3M= 2023-09-29T22:28:02.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UKvw2F98oLYcDpsoyBEdtCIiO3M=', 'Content-Length': '0'} 2023-09-29T22:28:02.784 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c43fbd4f584f7f9f-0065174f72-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.784 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-28 2023-09-29T22:28:02.784 DEBUG:boto:path=/dmuhsu-28/ 2023-09-29T22:28:02.784 DEBUG:boto:auth_path=/dmuhsu-28/ 2023-09-29T22:28:02.784 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.784 DEBUG:boto:Path: /dmuhsu-28/ 2023-09-29T22:28:02.784 DEBUG:boto:Data: 2023-09-29T22:28:02.785 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.785 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.785 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.785 DEBUG:boto:Params: {} 2023-09-29T22:28:02.785 DEBUG:boto:Token: None 2023-09-29T22:28:02.785 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-28/ 2023-09-29T22:28:02.785 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MBu/t7PkkVOql4Jfz3XRN5nN12s= 2023-09-29T22:28:02.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MBu/t7PkkVOql4Jfz3XRN5nN12s=', 'Content-Length': '0'} 2023-09-29T22:28:02.811 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4aa4236391b7bb3-0065174f72-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.811 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-29 2023-09-29T22:28:02.811 DEBUG:boto:path=/dmuhsu-29/ 2023-09-29T22:28:02.811 DEBUG:boto:auth_path=/dmuhsu-29/ 2023-09-29T22:28:02.811 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.811 DEBUG:boto:Path: /dmuhsu-29/ 2023-09-29T22:28:02.812 DEBUG:boto:Data: 2023-09-29T22:28:02.812 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.812 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.812 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.812 DEBUG:boto:Params: {} 2023-09-29T22:28:02.812 DEBUG:boto:Token: None 2023-09-29T22:28:02.812 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-29/ 2023-09-29T22:28:02.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r9o4r2iNSOzErWVGXyTk0r4PnzA= 2023-09-29T22:28:02.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r9o4r2iNSOzErWVGXyTk0r4PnzA=', 'Content-Length': '0'} 2023-09-29T22:28:02.835 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001bdb9a42ed6b3e97-0065174f72-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.836 DEBUG:boto:path=/dmuhsu-27/ 2023-09-29T22:28:02.836 DEBUG:boto:auth_path=/dmuhsu-27/ 2023-09-29T22:28:02.836 DEBUG:boto:path=/dmuhsu-27/?versioning 2023-09-29T22:28:02.836 DEBUG:boto:auth_path=/dmuhsu-27/?versioning 2023-09-29T22:28:02.836 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.836 DEBUG:boto:Path: /dmuhsu-27/?versioning 2023-09-29T22:28:02.836 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:28:02.836 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.836 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.836 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.837 DEBUG:boto:Params: {} 2023-09-29T22:28:02.837 DEBUG:boto:Token: None 2023-09-29T22:28:02.837 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-27/?versioning 2023-09-29T22:28:02.837 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X104PkEo/i9c23j4F7uuy8xlin0= 2023-09-29T22:28:02.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X104PkEo/i9c23j4F7uuy8xlin0=', 'Content-Length': '228'} 2023-09-29T22:28:02.841 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b97b209c9f4a8fd4-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.841 DEBUG:boto:path=/dmuhsu-27/ 2023-09-29T22:28:02.842 DEBUG:boto:auth_path=/dmuhsu-27/ 2023-09-29T22:28:02.842 DEBUG:boto:path=/dmuhsu-27/?versioning 2023-09-29T22:28:02.842 DEBUG:boto:auth_path=/dmuhsu-27/?versioning 2023-09-29T22:28:02.842 DEBUG:boto:Method: GET 2023-09-29T22:28:02.842 DEBUG:boto:Path: /dmuhsu-27/?versioning 2023-09-29T22:28:02.842 DEBUG:boto:Data: 2023-09-29T22:28:02.842 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.842 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.842 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.842 DEBUG:boto:Params: {} 2023-09-29T22:28:02.843 DEBUG:boto:Token: None 2023-09-29T22:28:02.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-27/?versioning 2023-09-29T22:28:02.843 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FcvT17tkP54zOTB0eQlGfS+sCDQ= 2023-09-29T22:28:02.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FcvT17tkP54zOTB0eQlGfS+sCDQ=', 'Content-Length': '0'} 2023-09-29T22:28:02.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ade4622c225f564-0065174f72-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.844 DEBUG:boto:EnabledDisabled 2023-09-29T22:28:02.844 DEBUG:boto:path=/dmuhsu-28/ 2023-09-29T22:28:02.845 DEBUG:boto:auth_path=/dmuhsu-28/ 2023-09-29T22:28:02.845 DEBUG:boto:path=/dmuhsu-28/?versioning 2023-09-29T22:28:02.845 DEBUG:boto:auth_path=/dmuhsu-28/?versioning 2023-09-29T22:28:02.845 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.845 DEBUG:boto:Path: /dmuhsu-28/?versioning 2023-09-29T22:28:02.845 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:28:02.845 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.845 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.845 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.845 DEBUG:boto:Params: {} 2023-09-29T22:28:02.846 DEBUG:boto:Token: None 2023-09-29T22:28:02.846 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-28/?versioning 2023-09-29T22:28:02.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:je131DpvIIPG5OMKxZ7UZ+ibaTY= 2023-09-29T22:28:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:je131DpvIIPG5OMKxZ7UZ+ibaTY=', 'Content-Length': '228'} 2023-09-29T22:28:02.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc59504225e70fb4-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.853 DEBUG:boto:path=/dmuhsu-28/ 2023-09-29T22:28:02.853 DEBUG:boto:auth_path=/dmuhsu-28/ 2023-09-29T22:28:02.853 DEBUG:boto:path=/dmuhsu-28/?versioning 2023-09-29T22:28:02.853 DEBUG:boto:auth_path=/dmuhsu-28/?versioning 2023-09-29T22:28:02.854 DEBUG:boto:Method: GET 2023-09-29T22:28:02.854 DEBUG:boto:Path: /dmuhsu-28/?versioning 2023-09-29T22:28:02.854 DEBUG:boto:Data: 2023-09-29T22:28:02.854 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.854 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.854 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.854 DEBUG:boto:Params: {} 2023-09-29T22:28:02.854 DEBUG:boto:Token: None 2023-09-29T22:28:02.854 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-28/?versioning 2023-09-29T22:28:02.855 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L443SjMC2ov+FVr2GHoXSUbMJnk= 2023-09-29T22:28:02.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L443SjMC2ov+FVr2GHoXSUbMJnk=', 'Content-Length': '0'} 2023-09-29T22:28:02.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d268031a0baa9bd3-0065174f72-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.898 DEBUG:boto:EnabledDisabled 2023-09-29T22:28:02.898 DEBUG:boto:path=/dmuhsu-29/ 2023-09-29T22:28:02.899 DEBUG:boto:auth_path=/dmuhsu-29/ 2023-09-29T22:28:02.899 DEBUG:boto:path=/dmuhsu-29/?versioning 2023-09-29T22:28:02.899 DEBUG:boto:auth_path=/dmuhsu-29/?versioning 2023-09-29T22:28:02.899 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.899 DEBUG:boto:Path: /dmuhsu-29/?versioning 2023-09-29T22:28:02.899 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:28:02.899 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.899 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.899 DEBUG:boto:Params: {} 2023-09-29T22:28:02.900 DEBUG:boto:Token: None 2023-09-29T22:28:02.900 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-29/?versioning 2023-09-29T22:28:02.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gwYbmo2LDOYmh2AQTQj/DVOLiSI= 2023-09-29T22:28:02.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gwYbmo2LDOYmh2AQTQj/DVOLiSI=', 'Content-Length': '228'} 2023-09-29T22:28:02.906 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b56cc04317e5907-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.906 DEBUG:boto:path=/dmuhsu-29/ 2023-09-29T22:28:02.907 DEBUG:boto:auth_path=/dmuhsu-29/ 2023-09-29T22:28:02.907 DEBUG:boto:path=/dmuhsu-29/?versioning 2023-09-29T22:28:02.907 DEBUG:boto:auth_path=/dmuhsu-29/?versioning 2023-09-29T22:28:02.907 DEBUG:boto:Method: GET 2023-09-29T22:28:02.907 DEBUG:boto:Path: /dmuhsu-29/?versioning 2023-09-29T22:28:02.907 DEBUG:boto:Data: 2023-09-29T22:28:02.907 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.907 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.907 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.907 DEBUG:boto:Params: {} 2023-09-29T22:28:02.908 DEBUG:boto:Token: None 2023-09-29T22:28:02.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-29/?versioning 2023-09-29T22:28:02.908 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:e0XSeWaBeSnJCTQZl3ZYhZygbLM= 2023-09-29T22:28:02.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:e0XSeWaBeSnJCTQZl3ZYhZygbLM=', 'Content-Length': '0'} 2023-09-29T22:28:02.909 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071ee9992078fd74f-0065174f72-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.950 DEBUG:boto:EnabledDisabled 2023-09-29T22:28:02.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-09-29T22:28:02.951 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-30 2023-09-29T22:28:02.951 DEBUG:boto:path=/dmuhsu-30/ 2023-09-29T22:28:02.952 DEBUG:boto:auth_path=/dmuhsu-30/ 2023-09-29T22:28:02.952 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.952 DEBUG:boto:Path: /dmuhsu-30/ 2023-09-29T22:28:02.952 DEBUG:boto:Data: 2023-09-29T22:28:02.952 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.952 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:02.952 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.952 DEBUG:boto:Params: {} 2023-09-29T22:28:02.952 DEBUG:boto:Token: None 2023-09-29T22:28:02.953 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-30/ 2023-09-29T22:28:02.953 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3is1kw8veY+2k76TGZCITf4STGQ= 2023-09-29T22:28:02.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3is1kw8veY+2k76TGZCITf4STGQ=', 'Content-Length': '0'} 2023-09-29T22:28:02.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4edff2f7664cf30-0065174f72-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.966 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-31 2023-09-29T22:28:02.966 DEBUG:boto:path=/dmuhsu-31/ 2023-09-29T22:28:02.966 DEBUG:boto:auth_path=/dmuhsu-31/ 2023-09-29T22:28:02.966 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.966 DEBUG:boto:Path: /dmuhsu-31/ 2023-09-29T22:28:02.966 DEBUG:boto:Data: 2023-09-29T22:28:02.967 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.967 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:02.967 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.967 DEBUG:boto:Params: {} 2023-09-29T22:28:02.967 DEBUG:boto:Token: None 2023-09-29T22:28:02.967 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-31/ 2023-09-29T22:28:02.967 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HVv3lY/bqIPmzPPcoieynfOwwlo= 2023-09-29T22:28:02.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HVv3lY/bqIPmzPPcoieynfOwwlo=', 'Content-Length': '0'} 2023-09-29T22:28:02.994 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000231a52559dbe3484-0065174f72-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.994 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-32 2023-09-29T22:28:02.994 DEBUG:boto:path=/dmuhsu-32/ 2023-09-29T22:28:02.994 DEBUG:boto:auth_path=/dmuhsu-32/ 2023-09-29T22:28:02.995 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.995 DEBUG:boto:Path: /dmuhsu-32/ 2023-09-29T22:28:02.995 DEBUG:boto:Data: 2023-09-29T22:28:02.995 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.995 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:02.995 DEBUG:boto:Port: 8001 2023-09-29T22:28:02.995 DEBUG:boto:Params: {} 2023-09-29T22:28:02.995 DEBUG:boto:Token: None 2023-09-29T22:28:02.995 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:02 GMT /dmuhsu-32/ 2023-09-29T22:28:02.996 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wSD31e+M9pnAJW9sNeDjqzwCYY4= 2023-09-29T22:28:02.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wSD31e+M9pnAJW9sNeDjqzwCYY4=', 'Content-Length': '0'} 2023-09-29T22:28:03.016 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000faae34699b3137e9-0065174f72-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.016 DEBUG:boto:path=/dmuhsu-30/ 2023-09-29T22:28:03.016 DEBUG:boto:auth_path=/dmuhsu-30/ 2023-09-29T22:28:03.016 DEBUG:boto:path=/dmuhsu-30/?acl 2023-09-29T22:28:03.016 DEBUG:boto:auth_path=/dmuhsu-30/?acl 2023-09-29T22:28:03.017 DEBUG:boto:Method: GET 2023-09-29T22:28:03.017 DEBUG:boto:Path: /dmuhsu-30/?acl 2023-09-29T22:28:03.017 DEBUG:boto:Data: 2023-09-29T22:28:03.017 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.017 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.017 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.017 DEBUG:boto:Params: {} 2023-09-29T22:28:03.017 DEBUG:boto:Token: None 2023-09-29T22:28:03.017 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-30/?acl 2023-09-29T22:28:03.018 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T53tZNLMt8tqSZQlIKli7Unk0nI= 2023-09-29T22:28:03.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T53tZNLMt8tqSZQlIKli7Unk0nI=', 'Content-Length': '0'} 2023-09-29T22:28:03.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000978165c4027d1d9a-0065174f73-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.020 DEBUG:boto:path=/dmuhsu-30/ 2023-09-29T22:28:03.020 DEBUG:boto:auth_path=/dmuhsu-30/ 2023-09-29T22:28:03.020 DEBUG:boto:path=/dmuhsu-30/?acl 2023-09-29T22:28:03.020 DEBUG:boto:auth_path=/dmuhsu-30/?acl 2023-09-29T22:28:03.020 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.020 DEBUG:boto:Path: /dmuhsu-30/?acl 2023-09-29T22:28:03.020 DEBUG:boto:Data: 2023-09-29T22:28:03.020 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:28:03.020 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.021 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.021 DEBUG:boto:Params: {} 2023-09-29T22:28:03.021 DEBUG:boto:Token: None 2023-09-29T22:28:03.021 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT x-amz-acl:public-read /dmuhsu-30/?acl 2023-09-29T22:28:03.021 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G9GNYNYEy1uUu65Uq61QXJM1xQc= 2023-09-29T22:28:03.021 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G9GNYNYEy1uUu65Uq61QXJM1xQc=', 'Content-Length': '0'} 2023-09-29T22:28:03.026 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e7413f59bc4adbb6-0065174f73-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.026 DEBUG:boto:path=/dmuhsu-30/ 2023-09-29T22:28:03.026 DEBUG:boto:auth_path=/dmuhsu-30/ 2023-09-29T22:28:03.026 DEBUG:boto:path=/dmuhsu-30/?acl 2023-09-29T22:28:03.026 DEBUG:boto:auth_path=/dmuhsu-30/?acl 2023-09-29T22:28:03.026 DEBUG:boto:Method: GET 2023-09-29T22:28:03.026 DEBUG:boto:Path: /dmuhsu-30/?acl 2023-09-29T22:28:03.026 DEBUG:boto:Data: 2023-09-29T22:28:03.027 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.027 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.027 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.027 DEBUG:boto:Params: {} 2023-09-29T22:28:03.027 DEBUG:boto:Token: None 2023-09-29T22:28:03.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-30/?acl 2023-09-29T22:28:03.027 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T53tZNLMt8tqSZQlIKli7Unk0nI= 2023-09-29T22:28:03.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T53tZNLMt8tqSZQlIKli7Unk0nI=', 'Content-Length': '0'} 2023-09-29T22:28:03.028 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064f35d02b6f2da03-0065174f73-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.071 DEBUG:boto:path=/dmuhsu-31/ 2023-09-29T22:28:03.071 DEBUG:boto:auth_path=/dmuhsu-31/ 2023-09-29T22:28:03.071 DEBUG:boto:path=/dmuhsu-31/?acl 2023-09-29T22:28:03.071 DEBUG:boto:auth_path=/dmuhsu-31/?acl 2023-09-29T22:28:03.071 DEBUG:boto:Method: GET 2023-09-29T22:28:03.071 DEBUG:boto:Path: /dmuhsu-31/?acl 2023-09-29T22:28:03.071 DEBUG:boto:Data: 2023-09-29T22:28:03.071 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.071 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:03.072 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.072 DEBUG:boto:Params: {} 2023-09-29T22:28:03.072 DEBUG:boto:Token: None 2023-09-29T22:28:03.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-31/?acl 2023-09-29T22:28:03.072 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kNNZfNKcqPiji3yuCX+fJVhE3Vs= 2023-09-29T22:28:03.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNNZfNKcqPiji3yuCX+fJVhE3Vs=', 'Content-Length': '0'} 2023-09-29T22:28:03.073 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5e25ea749e28f35-0065174f73-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.074 DEBUG:boto:path=/dmuhsu-31/ 2023-09-29T22:28:03.074 DEBUG:boto:auth_path=/dmuhsu-31/ 2023-09-29T22:28:03.074 DEBUG:boto:path=/dmuhsu-31/?acl 2023-09-29T22:28:03.074 DEBUG:boto:auth_path=/dmuhsu-31/?acl 2023-09-29T22:28:03.074 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.074 DEBUG:boto:Path: /dmuhsu-31/?acl 2023-09-29T22:28:03.075 DEBUG:boto:Data: 2023-09-29T22:28:03.075 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:28:03.075 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:03.075 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.075 DEBUG:boto:Params: {} 2023-09-29T22:28:03.075 DEBUG:boto:Token: None 2023-09-29T22:28:03.075 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT x-amz-acl:public-read /dmuhsu-31/?acl 2023-09-29T22:28:03.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WhwsBtz6fS3rUXfvu5ZTWIIdVZ8= 2023-09-29T22:28:03.075 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WhwsBtz6fS3rUXfvu5ZTWIIdVZ8=', 'Content-Length': '0'} 2023-09-29T22:28:03.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff6409ab03e1662d-0065174f73-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.082 DEBUG:boto:path=/dmuhsu-31/ 2023-09-29T22:28:03.083 DEBUG:boto:auth_path=/dmuhsu-31/ 2023-09-29T22:28:03.083 DEBUG:boto:path=/dmuhsu-31/?acl 2023-09-29T22:28:03.083 DEBUG:boto:auth_path=/dmuhsu-31/?acl 2023-09-29T22:28:03.083 DEBUG:boto:Method: GET 2023-09-29T22:28:03.083 DEBUG:boto:Path: /dmuhsu-31/?acl 2023-09-29T22:28:03.083 DEBUG:boto:Data: 2023-09-29T22:28:03.083 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.083 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:03.083 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.083 DEBUG:boto:Params: {} 2023-09-29T22:28:03.084 DEBUG:boto:Token: None 2023-09-29T22:28:03.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-31/?acl 2023-09-29T22:28:03.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kNNZfNKcqPiji3yuCX+fJVhE3Vs= 2023-09-29T22:28:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNNZfNKcqPiji3yuCX+fJVhE3Vs=', 'Content-Length': '0'} 2023-09-29T22:28:03.085 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ae2fe3a9db5a8d00-0065174f73-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.085 DEBUG:boto:path=/dmuhsu-32/ 2023-09-29T22:28:03.085 DEBUG:boto:auth_path=/dmuhsu-32/ 2023-09-29T22:28:03.085 DEBUG:boto:path=/dmuhsu-32/?acl 2023-09-29T22:28:03.086 DEBUG:boto:auth_path=/dmuhsu-32/?acl 2023-09-29T22:28:03.086 DEBUG:boto:Method: GET 2023-09-29T22:28:03.086 DEBUG:boto:Path: /dmuhsu-32/?acl 2023-09-29T22:28:03.086 DEBUG:boto:Data: 2023-09-29T22:28:03.086 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.086 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:03.086 DEBUG:boto:Port: 8001 2023-09-29T22:28:03.086 DEBUG:boto:Params: {} 2023-09-29T22:28:03.086 DEBUG:boto:Token: None 2023-09-29T22:28:03.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-32/?acl 2023-09-29T22:28:03.087 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CMOVX86/G2SMEcEejzcBLa2SCRo= 2023-09-29T22:28:03.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CMOVX86/G2SMEcEejzcBLa2SCRo=', 'Content-Length': '0'} 2023-09-29T22:28:03.088 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000948d9fdae9bfaa39-0065174f73-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.088 DEBUG:boto:path=/dmuhsu-32/ 2023-09-29T22:28:03.088 DEBUG:boto:auth_path=/dmuhsu-32/ 2023-09-29T22:28:03.089 DEBUG:boto:path=/dmuhsu-32/?acl 2023-09-29T22:28:03.089 DEBUG:boto:auth_path=/dmuhsu-32/?acl 2023-09-29T22:28:03.089 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.089 DEBUG:boto:Path: /dmuhsu-32/?acl 2023-09-29T22:28:03.089 DEBUG:boto:Data: 2023-09-29T22:28:03.089 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:28:03.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:03.089 DEBUG:boto:Port: 8001 2023-09-29T22:28:03.089 DEBUG:boto:Params: {} 2023-09-29T22:28:03.089 DEBUG:boto:Token: None 2023-09-29T22:28:03.090 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT x-amz-acl:public-read /dmuhsu-32/?acl 2023-09-29T22:28:03.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ke9GUj1B11GIWaSj3EbiapFADdM= 2023-09-29T22:28:03.090 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ke9GUj1B11GIWaSj3EbiapFADdM=', 'Content-Length': '0'} 2023-09-29T22:28:03.096 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ef382e96e62ab93-0065174f73-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.097 DEBUG:boto:path=/dmuhsu-32/ 2023-09-29T22:28:03.097 DEBUG:boto:auth_path=/dmuhsu-32/ 2023-09-29T22:28:03.097 DEBUG:boto:path=/dmuhsu-32/?acl 2023-09-29T22:28:03.097 DEBUG:boto:auth_path=/dmuhsu-32/?acl 2023-09-29T22:28:03.097 DEBUG:boto:Method: GET 2023-09-29T22:28:03.097 DEBUG:boto:Path: /dmuhsu-32/?acl 2023-09-29T22:28:03.097 DEBUG:boto:Data: 2023-09-29T22:28:03.097 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.097 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:03.098 DEBUG:boto:Port: 8001 2023-09-29T22:28:03.098 DEBUG:boto:Params: {} 2023-09-29T22:28:03.098 DEBUG:boto:Token: None 2023-09-29T22:28:03.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-32/?acl 2023-09-29T22:28:03.098 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CMOVX86/G2SMEcEejzcBLa2SCRo= 2023-09-29T22:28:03.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CMOVX86/G2SMEcEejzcBLa2SCRo=', 'Content-Length': '0'} 2023-09-29T22:28:03.099 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000399edba2725062df-0065174f73-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-09-29T22:28:03.100 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-33 2023-09-29T22:28:03.100 DEBUG:boto:path=/dmuhsu-33/ 2023-09-29T22:28:03.101 DEBUG:boto:auth_path=/dmuhsu-33/ 2023-09-29T22:28:03.101 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.101 DEBUG:boto:Path: /dmuhsu-33/ 2023-09-29T22:28:03.101 DEBUG:boto:Data: 2023-09-29T22:28:03.101 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.101 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.101 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.101 DEBUG:boto:Params: {} 2023-09-29T22:28:03.101 DEBUG:boto:Token: None 2023-09-29T22:28:03.102 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-33/ 2023-09-29T22:28:03.102 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dAT/TtMh6i+/fNWDAPM5RmWTnmc= 2023-09-29T22:28:03.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dAT/TtMh6i+/fNWDAPM5RmWTnmc=', 'Content-Length': '0'} 2023-09-29T22:28:03.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b8a8a895af0c625-0065174f73-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.113 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-34 2023-09-29T22:28:03.113 DEBUG:boto:path=/dmuhsu-34/ 2023-09-29T22:28:03.114 DEBUG:boto:auth_path=/dmuhsu-34/ 2023-09-29T22:28:03.114 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.114 DEBUG:boto:Path: /dmuhsu-34/ 2023-09-29T22:28:03.114 DEBUG:boto:Data: 2023-09-29T22:28:03.114 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.114 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:03.114 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.114 DEBUG:boto:Params: {} 2023-09-29T22:28:03.114 DEBUG:boto:Token: None 2023-09-29T22:28:03.115 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-34/ 2023-09-29T22:28:03.115 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JMxwcF5KY2rH8cBRFcCjdk+lAck= 2023-09-29T22:28:03.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JMxwcF5KY2rH8cBRFcCjdk+lAck=', 'Content-Length': '0'} 2023-09-29T22:28:03.182 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002efee22a8369d349-0065174f73-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.182 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-35 2023-09-29T22:28:03.182 DEBUG:boto:path=/dmuhsu-35/ 2023-09-29T22:28:03.182 DEBUG:boto:auth_path=/dmuhsu-35/ 2023-09-29T22:28:03.182 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.182 DEBUG:boto:Path: /dmuhsu-35/ 2023-09-29T22:28:03.183 DEBUG:boto:Data: 2023-09-29T22:28:03.183 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.183 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:03.183 DEBUG:boto:Port: 8001 2023-09-29T22:28:03.183 DEBUG:boto:Params: {} 2023-09-29T22:28:03.183 DEBUG:boto:Token: None 2023-09-29T22:28:03.183 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-35/ 2023-09-29T22:28:03.183 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kTh0HFWEf4BV0D0AGLxz458rYqA= 2023-09-29T22:28:03.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kTh0HFWEf4BV0D0AGLxz458rYqA=', 'Content-Length': '0'} 2023-09-29T22:28:03.229 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e64c455b998d1a0-0065174f73-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.229 DEBUG:boto:path=/dmuhsu-33/ 2023-09-29T22:28:03.229 DEBUG:boto:auth_path=/dmuhsu-33/ 2023-09-29T22:28:03.229 DEBUG:boto:path=/dmuhsu-33/?cors 2023-09-29T22:28:03.229 DEBUG:boto:auth_path=/dmuhsu-33/?cors 2023-09-29T22:28:03.229 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.230 DEBUG:boto:Path: /dmuhsu-33/?cors 2023-09-29T22:28:03.230 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:28:03.230 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:28:03.230 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.230 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.230 DEBUG:boto:Params: {} 2023-09-29T22:28:03.230 DEBUG:boto:Token: None 2023-09-29T22:28:03.230 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-33/?cors 2023-09-29T22:28:03.231 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cECXxdQjDFfCMTmc9YNwVr9VINY= 2023-09-29T22:28:03.231 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cECXxdQjDFfCMTmc9YNwVr9VINY=', 'Content-Length': '218'} 2023-09-29T22:28:03.243 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000250021cf43575c15-0065174f73-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.243 DEBUG:boto:path=/dmuhsu-33/ 2023-09-29T22:28:03.244 DEBUG:boto:auth_path=/dmuhsu-33/ 2023-09-29T22:28:03.244 DEBUG:boto:path=/dmuhsu-33/?cors 2023-09-29T22:28:03.244 DEBUG:boto:auth_path=/dmuhsu-33/?cors 2023-09-29T22:28:03.244 DEBUG:boto:Method: GET 2023-09-29T22:28:03.244 DEBUG:boto:Path: /dmuhsu-33/?cors 2023-09-29T22:28:03.244 DEBUG:boto:Data: 2023-09-29T22:28:03.244 DEBUG:boto:Headers: {} 2023-09-29T22:28:03.244 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:03.244 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.244 DEBUG:boto:Params: {} 2023-09-29T22:28:03.245 DEBUG:boto:Token: None 2023-09-29T22:28:03.245 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-33/?cors 2023-09-29T22:28:03.245 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/sEbY2SfuFsHvJWEWFBs1kwiDn8= 2023-09-29T22:28:03.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/sEbY2SfuFsHvJWEWFBs1kwiDn8=', 'Content-Length': '0'} 2023-09-29T22:28:03.246 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e02548f7df29278-0065174f73-1104-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:03.246 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:28:03.246 DEBUG:boto:path=/dmuhsu-34/ 2023-09-29T22:28:03.247 DEBUG:boto:auth_path=/dmuhsu-34/ 2023-09-29T22:28:03.247 DEBUG:boto:path=/dmuhsu-34/?cors 2023-09-29T22:28:03.247 DEBUG:boto:auth_path=/dmuhsu-34/?cors 2023-09-29T22:28:03.247 DEBUG:boto:Method: PUT 2023-09-29T22:28:03.247 DEBUG:boto:Path: /dmuhsu-34/?cors 2023-09-29T22:28:03.247 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:28:03.247 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:28:03.247 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:03.247 DEBUG:boto:Port: 8000 2023-09-29T22:28:03.247 DEBUG:boto:Params: {} 2023-09-29T22:28:03.248 DEBUG:boto:Token: None 2023-09-29T22:28:03.248 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:28:03 GMT /dmuhsu-34/?cors 2023-09-29T22:28:03.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eUhf9hGFmc6WZaBs7nhNfuwiAMk= 2023-09-29T22:28:03.248 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eUhf9hGFmc6WZaBs7nhNfuwiAMk=', 'Content-Length': '218'} 2023-09-29T22:28:04.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa26b265fc7c7f34-0065174f73-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:04.256 DEBUG:boto:path=/dmuhsu-34/ 2023-09-29T22:28:04.256 DEBUG:boto:auth_path=/dmuhsu-34/ 2023-09-29T22:28:04.256 DEBUG:boto:path=/dmuhsu-34/?cors 2023-09-29T22:28:04.256 DEBUG:boto:auth_path=/dmuhsu-34/?cors 2023-09-29T22:28:04.256 DEBUG:boto:Method: GET 2023-09-29T22:28:04.256 DEBUG:boto:Path: /dmuhsu-34/?cors 2023-09-29T22:28:04.257 DEBUG:boto:Data: 2023-09-29T22:28:04.257 DEBUG:boto:Headers: {} 2023-09-29T22:28:04.257 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:04.257 DEBUG:boto:Port: 8000 2023-09-29T22:28:04.257 DEBUG:boto:Params: {} 2023-09-29T22:28:04.257 DEBUG:boto:Token: None 2023-09-29T22:28:04.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:04 GMT /dmuhsu-34/?cors 2023-09-29T22:28:04.257 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H9njAkRUdT2cykiL++KzRA47kCs= 2023-09-29T22:28:04.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H9njAkRUdT2cykiL++KzRA47kCs=', 'Content-Length': '0'} 2023-09-29T22:28:04.259 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5b53591c0cf005f-0065174f74-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:04.259 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:28:04.260 DEBUG:boto:path=/dmuhsu-35/ 2023-09-29T22:28:04.260 DEBUG:boto:auth_path=/dmuhsu-35/ 2023-09-29T22:28:04.260 DEBUG:boto:path=/dmuhsu-35/?cors 2023-09-29T22:28:04.260 DEBUG:boto:auth_path=/dmuhsu-35/?cors 2023-09-29T22:28:04.260 DEBUG:boto:Method: PUT 2023-09-29T22:28:04.260 DEBUG:boto:Path: /dmuhsu-35/?cors 2023-09-29T22:28:04.260 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:28:04.261 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:28:04.261 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:04.261 DEBUG:boto:Port: 8001 2023-09-29T22:28:04.261 DEBUG:boto:Params: {} 2023-09-29T22:28:04.261 DEBUG:boto:Token: None 2023-09-29T22:28:04.261 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:28:04 GMT /dmuhsu-35/?cors 2023-09-29T22:28:04.261 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S2sgJR1wdX2yu9DlJ/Am7UApRYY= 2023-09-29T22:28:04.262 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S2sgJR1wdX2yu9DlJ/Am7UApRYY=', 'Content-Length': '218'} 2023-09-29T22:28:05.269 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bdb30559f17f69c9-0065174f74-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:05.269 DEBUG:boto:path=/dmuhsu-35/ 2023-09-29T22:28:05.269 DEBUG:boto:auth_path=/dmuhsu-35/ 2023-09-29T22:28:05.269 DEBUG:boto:path=/dmuhsu-35/?cors 2023-09-29T22:28:05.270 DEBUG:boto:auth_path=/dmuhsu-35/?cors 2023-09-29T22:28:05.270 DEBUG:boto:Method: GET 2023-09-29T22:28:05.270 DEBUG:boto:Path: /dmuhsu-35/?cors 2023-09-29T22:28:05.270 DEBUG:boto:Data: 2023-09-29T22:28:05.270 DEBUG:boto:Headers: {} 2023-09-29T22:28:05.270 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:05.270 DEBUG:boto:Port: 8001 2023-09-29T22:28:05.270 DEBUG:boto:Params: {} 2023-09-29T22:28:05.271 DEBUG:boto:Token: None 2023-09-29T22:28:05.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:05 GMT /dmuhsu-35/?cors 2023-09-29T22:28:05.271 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VSJFLBN5dj1Q+XhtT3wemKMLOwE= 2023-09-29T22:28:05.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VSJFLBN5dj1Q+XhtT3wemKMLOwE=', 'Content-Length': '0'} 2023-09-29T22:28:05.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001841fef2c88a65c6-0065174f75-110a-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:05.272 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:28:05.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-09-29T22:28:05.273 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-36 2023-09-29T22:28:05.273 DEBUG:boto:path=/dmuhsu-36/ 2023-09-29T22:28:05.273 DEBUG:boto:auth_path=/dmuhsu-36/ 2023-09-29T22:28:05.274 DEBUG:boto:Method: PUT 2023-09-29T22:28:05.274 DEBUG:boto:Path: /dmuhsu-36/ 2023-09-29T22:28:05.274 DEBUG:boto:Data: 2023-09-29T22:28:05.274 DEBUG:boto:Headers: {} 2023-09-29T22:28:05.274 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:28:05.274 DEBUG:boto:Port: 8000 2023-09-29T22:28:05.274 DEBUG:boto:Params: {} 2023-09-29T22:28:05.274 DEBUG:boto:Token: None 2023-09-29T22:28:05.275 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:05 GMT /dmuhsu-36/ 2023-09-29T22:28:05.275 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SreNg293zZtVH9eeArZXWkXP8tg= 2023-09-29T22:28:05.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SreNg293zZtVH9eeArZXWkXP8tg=', 'Content-Length': '0'} 2023-09-29T22:28:05.287 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a2aefa3e90c8aac-0065174f75-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:05.287 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-37 2023-09-29T22:28:05.287 DEBUG:boto:path=/dmuhsu-37/ 2023-09-29T22:28:05.287 DEBUG:boto:auth_path=/dmuhsu-37/ 2023-09-29T22:28:05.287 DEBUG:boto:Method: PUT 2023-09-29T22:28:05.287 DEBUG:boto:Path: /dmuhsu-37/ 2023-09-29T22:28:05.287 DEBUG:boto:Data: 2023-09-29T22:28:05.287 DEBUG:boto:Headers: {} 2023-09-29T22:28:05.288 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:28:05.288 DEBUG:boto:Port: 8000 2023-09-29T22:28:05.288 DEBUG:boto:Params: {} 2023-09-29T22:28:05.288 DEBUG:boto:Token: None 2023-09-29T22:28:05.288 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:05 GMT /dmuhsu-37/ 2023-09-29T22:28:05.288 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:U6hTIqSJfX9k9sipQPJDHBxwqGI= 2023-09-29T22:28:05.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U6hTIqSJfX9k9sipQPJDHBxwqGI=', 'Content-Length': '0'} 2023-09-29T22:28:05.335 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fa8ce65286a0c60-0065174f75-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:05.335 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-38 2023-09-29T22:28:05.335 DEBUG:boto:path=/dmuhsu-38/ 2023-09-29T22:28:05.336 DEBUG:boto:auth_path=/dmuhsu-38/ 2023-09-29T22:28:05.336 DEBUG:boto:Method: PUT 2023-09-29T22:28:05.336 DEBUG:boto:Path: /dmuhsu-38/ 2023-09-29T22:28:05.336 DEBUG:boto:Data: 2023-09-29T22:28:05.336 DEBUG:boto:Headers: {} 2023-09-29T22:28:05.336 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:28:05.336 DEBUG:boto:Port: 8001 2023-09-29T22:28:05.336 DEBUG:boto:Params: {} 2023-09-29T22:28:05.337 DEBUG:boto:Token: None 2023-09-29T22:28:05.337 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:28:05 GMT /dmuhsu-38/ 2023-09-29T22:28:05.337 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bPr54LNQJVl4OQY2mW5OFJ9aRoM= 2023-09-29T22:28:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bPr54LNQJVl4OQY2mW5OFJ9aRoM=', 'Content-Length': '0'} 2023-09-29T22:28:05.357 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a025d187a533161-0065174f75-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:05.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:05.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:05.358 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:05.416 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:05.416 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:05.477 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:28:05.489 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:28:05.489 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:05.490 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.350874_251.1", 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:28:05.350874Z" 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:05.491 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:28:05.492 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:05.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:28:05.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:05.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:05.493 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:05.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:05.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:05.623 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:05.623+0000 7fb0b589d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:05.624 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:05.624+0000 7fb0b589d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:05.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:28:05.636 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:28:05.636 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:28:05.636 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:28:05.636 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:05.637 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:05.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026482.903428_202.1", 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:05.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:05.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:05.642 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:05.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:05.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:28:05.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:28:05.644 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:28:05.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:05.644 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:28:05Z" 2023-09-29T22:28:05.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:28:05.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.903428_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.903428Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:05Z" } 2023-09-29T22:28:05.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:05.645 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:05.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:05.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-29T22:28:10.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:10.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:10.647 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:10.699 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:10.700 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:10.731 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:10.732+0000 7f51baf2d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:10.732 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:28:10.733+0000 7f51baf2d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:10.745 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:28:10.745 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:28:10.745 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:28:10.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:28:10.745 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:10.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:28:10.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:10.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026485.350874_251.1", 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:28:05.350874Z", 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:28:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:28:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:28:10Z" 2023-09-29T22:28:10.752 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:28:10.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.350874_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.350874Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:10Z" } 2023-09-29T22:28:10.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.350874_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.350874Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:10.753 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:10.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:10.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:28:10.753 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:28:10.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:10.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:10.754 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:10.801 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:10.827 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:10.827+0000 7f073f14b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:10.828 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:10.828+0000 7f073f14b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:10.839 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:10.839 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:10.839 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:10.840 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:10.841 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:10.842 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:10.843 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:10.844 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:10.845 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:10.846 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:10.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:10.846 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:10Z" 2023-09-29T22:28:10.846 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:10.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:10Z" } 2023-09-29T22:28:10.847 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:10.847 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:10.847 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:10.847 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:28:15.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:15.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:15.849 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:15.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:15.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:15.925 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:15.925+0000 7fbb8fcc0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:15.926 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:15.926+0000 7fbb8fcc0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:15.937 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:15.937 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:15.937 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:15.937 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:15.937 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:15.938 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:15.939 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:15.940 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.962427_205.1", 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:15.942 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:15Z" 2023-09-29T22:28:15.943 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:15.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.962427_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.962427Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:15Z" } 2023-09-29T22:28:15.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:15.945 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:15.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:15.945 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.962427_205.1 2023-09-29T22:28:20.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:20.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:20.946 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:20.995 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:20.995 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:21.025 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:21.025+0000 7f15c6730880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:21.026 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:21.026+0000 7f15c6730880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:21.038 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:21.039 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:21.040 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:21Z" 2023-09-29T22:28:21.044 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:21.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:21Z" } 2023-09-29T22:28:21.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:21.045 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:21.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:21.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-29T22:28:26.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:26.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:26.047 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:26.117 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:26.117 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:26.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:26.154+0000 7f2f54332880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:26.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:26.155+0000 7f2f54332880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:26.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:26.168 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:26.169 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:28:26.170 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:26.171 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.172 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:26.173 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:26.173 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:26.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:26.173 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:26Z" 2023-09-29T22:28:26.173 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:26.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:26Z" } 2023-09-29T22:28:26.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:26.174 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:26.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:26.174 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-29T22:28:31.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:31.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:31.176 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:31.221 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:31.221 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:31.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:31.247+0000 7f07af026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:31.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:31.248+0000 7f07af026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:31.260 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:31.260 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:31.261 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:31.262 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:31.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:31.265 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:31Z" 2023-09-29T22:28:31.266 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:31.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:31Z" } 2023-09-29T22:28:31.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:31.268 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:31.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:31.268 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:28:36.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:36.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:36.270 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:36.331 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:36.332 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:36.359 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:36.359+0000 7f2425f37880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:36.360 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:36.360+0000 7f2425f37880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:36.373 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:36.374 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:36.375 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:36.376 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:36.377 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:36.378 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:36Z" 2023-09-29T22:28:36.379 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:36.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:36Z" } 2023-09-29T22:28:36.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:36.380 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:36.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:36.381 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-29T22:28:41.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:41.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:41.382 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:41.461 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:41.461 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:41.496 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:41.495+0000 7f020d727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:41.497 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:41.496+0000 7f020d727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:41.508 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:41.508 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:41.508 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:41.509 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:41.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.825442_197.1", 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:41.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:41.513 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:41Z" 2023-09-29T22:28:41.514 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:41.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.825442_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.825442Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:41Z" } 2023-09-29T22:28:41.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:41.516 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:41.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:41.516 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.825442_197.1 2023-09-29T22:28:46.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:46.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:46.517 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:46.563 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:46.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:46.590+0000 7fee8c88b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:46.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:46.591+0000 7fee8c88b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:46.602 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:46.602 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:46.603 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:46.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:46.605 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:46.606 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:46.607 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:46.608 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:46.609 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:46.609 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:46Z" 2023-09-29T22:28:46.609 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:46.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:46Z" } 2023-09-29T22:28:46.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:46.610 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:46.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:46.610 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-29T22:28:51.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:51.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:51.612 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:51.659 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:51.659 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:51.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:51.687+0000 7f197251e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:51.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:51.687+0000 7f197251e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:51.698 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:51.698 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:51.698 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:51.699 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:51.700 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:51.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:28:51.702 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:51.703 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:51Z" 2023-09-29T22:28:51.704 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:51.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:51Z" } 2023-09-29T22:28:51.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:51.706 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:51.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:51.706 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:28:56.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:56.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:28:56.708 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:28:56.754 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:56.754 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:56.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:56.782+0000 7fb76a92c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:56.782 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:28:56.783+0000 7fb76a92c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:56.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:28:56.795 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:56.796 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:56.797 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:28:56.798 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:28:56.799 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:28:56.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:28:56.800 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:28:56Z" 2023-09-29T22:28:56.800 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:28:56.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:28:56Z" } 2023-09-29T22:28:56.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:28:56.801 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:56.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:28:56.801 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:29:01.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:01.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:01.803 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:01.850 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:01.850 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:01.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:01.878+0000 7f3f66c12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:01.878 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:01.879+0000 7f3f66c12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:01.890 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:01.890 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:01.891 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:01.892 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:01.893 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.905539_203.1", 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:01.894 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:01.895 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:01Z" 2023-09-29T22:29:01.896 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:01.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.905539_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.905539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:01Z" } 2023-09-29T22:29:01.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:01.898 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:01.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:01.898 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.905539_203.1 2023-09-29T22:29:06.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:06.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:06.900 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:06.971 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:06.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:07.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:07.017+0000 7fb5f8b30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:07.017 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:07.018+0000 7fb5f8b30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:07.029 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:07.029 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:07.029 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:07.030 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:07.031 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:07.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:07.033 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:07.034 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:07Z" 2023-09-29T22:29:07.035 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:07.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:07Z" } 2023-09-29T22:29:07.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:07.037 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:07.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:07.037 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:29:12.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:12.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:12.038 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:12.084 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:12.084 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:12.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:12.115+0000 7f703cd2c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:12.115 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:12.116+0000 7f703cd2c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:12.128 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:12.128 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:12.128 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:12.129 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:12.130 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.131 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:12.132 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:12.133 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:12.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:12.135 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:12Z" 2023-09-29T22:29:12.135 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:12.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:12Z" } 2023-09-29T22:29:12.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:12.136 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:12.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:12.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-29T22:29:17.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:17.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:17.138 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:17.185 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:17.185 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:17.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:17.213+0000 7fe56594b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:17.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:17.214+0000 7fe56594b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.962427_205.1", 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:17.230 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:17Z" 2023-09-29T22:29:17.231 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:17.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.962427_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.962427Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:17Z" } 2023-09-29T22:29:17.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:17.233 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:17.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:17.233 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.962427_205.1 2023-09-29T22:29:22.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:22.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:22.234 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:22.300 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:22.300 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:22.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:22.366+0000 7fd84591f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:22.367 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:22.367+0000 7fd84591f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:22.378 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:22.379 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:22.380 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:22.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:22.382 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.903428_202.1", 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:22.384 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:22.385 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:22.386 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:22.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:22.387 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:22Z" 2023-09-29T22:29:22.387 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:22.388 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.903428_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.903428Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:22Z" } 2023-09-29T22:29:22.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:22.388 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:22.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:22.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-29T22:29:27.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:27.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:27.389 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:27.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:27.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:27.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:27.473+0000 7f7031526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:27.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:27.474+0000 7f7031526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:27.486 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:27.487 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:27.489 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:27.490 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:27.492 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:27.493 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:27.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:27.494 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:27Z" 2023-09-29T22:29:27.494 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:27.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:27Z" } 2023-09-29T22:29:27.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:27.495 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:27.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:27.495 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-29T22:29:32.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:32.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:32.498 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:32.546 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:32.546 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:32.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:32.574+0000 7f4d52175880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:32.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:32.575+0000 7f4d52175880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:32.587 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:32.591 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:32.592 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:32.593 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:32.594 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:32.594 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:32Z" 2023-09-29T22:29:32.594 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:32.595 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:32Z" } 2023-09-29T22:29:32.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:32.595 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:32.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:32.595 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:29:37.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:37.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:37.597 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:37.643 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:37.643 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:37.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:37.669+0000 7f2c56c0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:37.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:37.670+0000 7f2c56c0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:37.682 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:37.682 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:37.682 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:37.682 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:37.683 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:37.684 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:37.686 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:37.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:37Z" 2023-09-29T22:29:37.688 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:37.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:37Z" } 2023-09-29T22:29:37.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:37.690 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:37.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:37.690 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:29:42.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:42.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:42.692 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:42.765 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:42.765 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:42.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:42.793+0000 7f5f5b723880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:42.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:42.794+0000 7f5f5b723880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:42.807 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:42.807 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:42.807 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:42.808 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:42.809 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.810 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:42.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:42.812 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:42Z" 2023-09-29T22:29:42.813 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:42.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:42Z" } 2023-09-29T22:29:42.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:42.815 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:42.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:42.815 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-29T22:29:47.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:47.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:47.816 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:47.894 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:47.894 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:47.937 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:47.937+0000 7f0d6cd27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:47.938 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:47.938+0000 7f0d6cd27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:47.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:47.951 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:47.952 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:47.954 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:47.955 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:47Z" 2023-09-29T22:29:47.956 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:47.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:47Z" } 2023-09-29T22:29:47.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:47.958 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:47.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:47.958 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-29T22:29:52.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:52.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:52.960 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:53.008 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:53.009 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:53.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:53.035+0000 7f9d72d26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:53.035 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:53.036+0000 7f9d72d26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:53.047 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:53.048 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:53.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:53.050 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:53.051 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:53.052 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:53.053 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:53.054 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:53Z" 2023-09-29T22:29:53.055 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:53.056 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:53Z" } 2023-09-29T22:29:53.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:53.057 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:53.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:53.057 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-29T22:29:58.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:58.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:29:58.059 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:29:58.105 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:29:58.105 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:29:58.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:58.135+0000 7f3ad1611880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:58.135 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:29:58.135+0000 7f3ad1611880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:29:58.147 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:29:58.147 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:29:58.147 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:29:58.147 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:29:58.147 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:58.148 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:58.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:58.150 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:58.151 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:29:58.151 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:58.151 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:58.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:58.152 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:29:58.153 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:58.154 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:29:58.155 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:29:58Z" 2023-09-29T22:29:58.156 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:29:58.156 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:29:58Z" } 2023-09-29T22:29:58.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:29:58.157 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:58.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:29:58.157 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-29T22:30:03.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:03.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:03.158 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:03.203 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:03.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:03.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:03.231+0000 7f667a927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:03.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:03.232+0000 7f667a927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:03.243 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:03.243 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:03.243 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:03.243 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:03.243 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:03.244 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:03.245 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:03.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:03.247 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:03.248 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:03.249 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:03.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:03.250 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:03Z" 2023-09-29T22:30:03.250 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:03.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:03Z" } 2023-09-29T22:30:03.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:03.251 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:03.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:03.251 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-29T22:30:08.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:08.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:08.253 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:08.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:08.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:08.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:08.332+0000 7fef5d323880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:08.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:08.332+0000 7fef5d323880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:08.344 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:08.344 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:08.344 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.345 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:08.346 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:08.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:08.348 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:08.348 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:08.348 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:08.348 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:08.349 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:30:08.350 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:08.352 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:08.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:08Z" 2023-09-29T22:30:08.354 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:08.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:08Z" } 2023-09-29T22:30:08.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:08.356 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:08.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:08.356 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-29T22:30:13.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:13.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:13.357 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:13.403 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:13.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:13.431+0000 7fa290f36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:13.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:13.431+0000 7fa290f36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:13.442 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:13.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:13.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:13.445 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:13.446 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:13.447 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:13.448 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:13.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:13.449 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:13Z" 2023-09-29T22:30:13.449 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:13.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:13Z" } 2023-09-29T22:30:13.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:13.450 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:13.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:13.450 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:30:18.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:18.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:18.452 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:18.499 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:18.499 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:18.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:18.526+0000 7effe471f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:18.527 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:18.527+0000 7effe471f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:18.538 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:18.538 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:18.539 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:18.540 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:18.541 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:18.542 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.543 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:18.544 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:18.545 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:18Z" 2023-09-29T22:30:18.546 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:18.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:18Z" } 2023-09-29T22:30:18.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:18.547 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:18.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:18.547 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:30:23.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:23.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:23.549 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:23.622 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:23.622 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:23.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:23.651+0000 7f715520f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:23.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:23.652+0000 7f715520f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:23.664 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:23.664 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:23.664 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:23.664 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:23.664 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:23.665 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:23.666 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:23.666 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:23.666 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:23.666 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:23.666 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:23.668 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:23.669 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.963214_206.1", 2023-09-29T22:30:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:23.670 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:23.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:23.673 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:23.674 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:23Z" 2023-09-29T22:30:23.675 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:23.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.963214_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.963214Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:23Z" } 2023-09-29T22:30:23.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.963214_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.963214Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:23.677 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:23.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.963214_206.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:23.677 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.963214_206.1 2023-09-29T22:30:28.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:28.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:28.678 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:28.726 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:28.754 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:28.753+0000 7fafeab26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:28.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:28.754+0000 7fafeab26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:28.766 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:28.767 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:28.768 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.768 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:28.768 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:28.768 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:28.768 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:28.769 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:28.770 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.771 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:28.772 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:28.773 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:28.774 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:28Z" 2023-09-29T22:30:28.775 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:28.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:28Z" } 2023-09-29T22:30:28.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:28.777 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:28.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:28.777 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-29T22:30:33.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:33.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:33.778 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:33.831 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:33.831 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:33.861+0000 7f98ac82e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:33.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:33.862+0000 7f98ac82e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:33.875 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:33.875 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:33.875 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006025_213.1", 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:33.880 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:33.881 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:33.882 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:33Z" 2023-09-29T22:30:33.883 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:33.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006025_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006025Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:33Z" } 2023-09-29T22:30:33.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006025_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:33.884 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:33.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006025_213.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:33.884 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.006025_213.1 2023-09-29T22:30:38.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:38.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:38.886 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:38.933 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:38.933 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:38.962 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:38.961+0000 7fe889b34880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:38.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:38.962+0000 7fe889b34880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:38.978 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:38.978 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:38.978 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:38.978 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:38.979 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.960282_204.1", 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:38.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:38.983 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:38Z" 2023-09-29T22:30:38.984 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:38.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.960282_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.960282Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:38Z" } 2023-09-29T22:30:38.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:38.986 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:38.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:38.986 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.960282_204.1 2023-09-29T22:30:43.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:43.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:43.988 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:44.040 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:44.040 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:44.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:44.070+0000 7f7b9f321880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:44.071 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:44.071+0000 7f7b9f321880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:44.084 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:44.084 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:44.084 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:44.084 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:44.084 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:44.085 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:44.086 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:44.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:44.089 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:44.090 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:44Z" 2023-09-29T22:30:44.091 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:44.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:44Z" } 2023-09-29T22:30:44.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:44.092 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:44.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:44.092 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-29T22:30:49.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:49.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:49.093 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:49.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:49.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:49.177 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:49.177+0000 7f0db354b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:49.178 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:49.178+0000 7f0db354b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:49.191 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:49.191 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:49.191 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:49.191 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:49.192 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:49.193 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:49.195 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:49.196 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:49Z" 2023-09-29T22:30:49.197 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:49.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:49Z" } 2023-09-29T22:30:49.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:49.199 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:49.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:49.199 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:30:54.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:54.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:54.201 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:54.248 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:54.248 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:54.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:54.277+0000 7f3e1d533880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:54.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:54.278+0000 7f3e1d533880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:54.289 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:54.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:54.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:54.292 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:54.293 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:54.294 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:54.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:54.296 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:54Z" 2023-09-29T22:30:54.296 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:54.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:54Z" } 2023-09-29T22:30:54.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:54.297 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:54.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:54.297 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-29T22:30:59.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:59.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:30:59.299 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:30:59.348 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:30:59.349 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:30:59.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:59.378+0000 7fa8a7c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:59.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:30:59.379+0000 7fa8a7c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:30:59.390 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.391 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:30:59.392 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:59.393 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:59.394 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:30:59.395 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:59.396 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:30:59Z" 2023-09-29T22:30:59.397 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:30:59.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:30:59Z" } 2023-09-29T22:30:59.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:30:59.399 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:59.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:30:59.399 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-29T22:31:04.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:04.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:04.400 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:04.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:04.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:04.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:04.533+0000 7ff7b0d2f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:04.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:04.539+0000 7ff7b0d2f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:04.561 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:04.562 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:04.563 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:04.564 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:31:04.565 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:04.566 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:04.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:04.568 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:04Z" 2023-09-29T22:31:04.569 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:04.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:04Z" } 2023-09-29T22:31:04.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:04.571 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:04.571 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:04.571 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-29T22:31:09.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:09.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:09.575 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:09.628 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:09.628 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:09.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:09.657+0000 7f9eb074b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:09.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:09.658+0000 7f9eb074b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:09.670 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:09.670 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:09.671 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:09.672 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:09.673 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:09.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:09.675 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:09Z" 2023-09-29T22:31:09.676 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:09.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:09Z" } 2023-09-29T22:31:09.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:09.678 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:09.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:09.678 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-29T22:31:14.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:14.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:14.680 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:14.732 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:14.733 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:14.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:14.763+0000 7fce4e1dc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:14.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:14.764+0000 7fce4e1dc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:14.776 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:14.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:14.778 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:14.779 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:14.780 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.003827_212.1", 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:14.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:14.782 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:14.783 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:14Z" 2023-09-29T22:31:14.784 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:14.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.003827_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.003827Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:14Z" } 2023-09-29T22:31:14.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:14.785 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:14.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:14.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.003827_212.1 2023-09-29T22:31:19.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:19.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:19.787 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:19.835 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:19.835 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:19.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:19.861+0000 7f7d3b40a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:19.862 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:19.862+0000 7f7d3b40a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:19.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:19.875 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:19.876 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:19.877 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:19.879 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:19Z" 2023-09-29T22:31:19.880 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:19.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:19Z" } 2023-09-29T22:31:19.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:19.882 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:19.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:19.882 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:31:24.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.883 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:24.938 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:24.938 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:24.969 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:24.969+0000 7f9b6dd33880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:24.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:24.970+0000 7f9b6dd33880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:24.983 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:24.983 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:24.983 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:24.984 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:24.985 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:24.986 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:24.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:24.988 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:24Z" 2023-09-29T22:31:24.989 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:24.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:24Z" } 2023-09-29T22:31:24.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:24.991 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:24.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:24.991 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-29T22:31:29.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:29.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:29.992 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:30.045 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:30.045 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:30.077 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:30.076+0000 7f38f66bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:30.078 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:30.077+0000 7f38f66bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:30.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:30.092 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:30.093 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:31:30.094 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:30.095 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:30.096 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:30Z" 2023-09-29T22:31:30.097 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:30.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:30Z" } 2023-09-29T22:31:30.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:30.098 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:30.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:30.099 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-29T22:31:35.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:35.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:35.100 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:35.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:35.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:35.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:35.173+0000 7fdb1e136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:35.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:35.174+0000 7fdb1e136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:35.186 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:35.186 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:35.186 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:35.187 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:35.188 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:35.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:35.190 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:31:35.197 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:35.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:35.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:35.197 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:35.198 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:35.199 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:35Z" 2023-09-29T22:31:35.200 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:35.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:35Z" } 2023-09-29T22:31:35.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:35.201 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:35.201 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:35.201 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-29T22:31:40.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:40.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:40.202 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:40.253 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:40.253 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:40.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:40.283+0000 7f89a520f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:40.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:40.284+0000 7f89a520f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:40.298 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:40.299 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:40.300 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:40.301 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:40.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:40.303 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:40.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:40.305 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:40Z" 2023-09-29T22:31:40.305 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:40.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:40Z" } 2023-09-29T22:31:40.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:40.306 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:40.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:40.306 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.965243_207.1 2023-09-29T22:31:45.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:45.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:45.308 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:45.384 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:45.385 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:45.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:45.441+0000 7f6dd8526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:45.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:45.442+0000 7f6dd8526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:45.454 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.455 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:45.456 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:45.457 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:31:45.458 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:45.459 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.460 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:45.461 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:45.461 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:45.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:45.461 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:45Z" 2023-09-29T22:31:45.461 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:45.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:45Z" } 2023-09-29T22:31:45.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:45.462 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:45.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:45.463 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-29T22:31:50.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:50.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:50.464 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:50.510 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:50.511 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:50.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:50.538+0000 7f08e0c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:50.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:50.539+0000 7f08e0c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:50.550 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.551 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:50.552 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:50.553 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:50.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:31:50.555 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:50.556 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.557 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:50.558 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:50.558 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:50.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:50.558 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:50Z" 2023-09-29T22:31:50.558 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:50.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:50Z" } 2023-09-29T22:31:50.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:50.559 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:50.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:50.560 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:31:55.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:55.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:55.561 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:55.613 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:55.613 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:55.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:55.645+0000 7fd8e4736880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:55.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:31:55.646+0000 7fd8e4736880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:55.658 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.659 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:31:55.660 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:55.661 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:55.662 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:31:55.663 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.664 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:31:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:31:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:31:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:31:55.665 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:31:55Z" 2023-09-29T22:31:55.665 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:31:55.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:55Z" } 2023-09-29T22:31:55.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:31:55.666 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:55.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:31:55.666 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-29T22:32:00.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:00.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:00.668 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:00.715 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:00.716 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:00.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:00.743+0000 7f5c1374b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:00.743 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:00.744+0000 7f5c1374b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:00.755 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:00.755 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:00.755 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:00.756 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.757 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:00.758 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.023985_216.1", 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-29T22:32:00.759 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:00.760 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:00.761 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:00.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:00.762 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:00Z" 2023-09-29T22:32:00.762 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:00.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.023985_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.023985Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:00Z" } 2023-09-29T22:32:00.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:00.763 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:00.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:00.763 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.023985_216.1 2023-09-29T22:32:05.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:05.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:05.765 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:05.817 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:05.818 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:05.849 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:05.849+0000 7f6c31321880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:05.850 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:05.850+0000 7f6c31321880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:05.862 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:05.863 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:05.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:05.865 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:05.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:05.867 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:05.868 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:05.869 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:05.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:05.869 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:05Z" 2023-09-29T22:32:05.869 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:05.870 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:05Z" } 2023-09-29T22:32:05.870 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:05.870 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:05.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:05.870 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:32:10.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:10.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:10.872 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:10.925 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:10.925 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:10.957 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:10.957+0000 7f190911f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:10.963 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:10.963+0000 7f190911f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:11.035 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.036 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:11.037 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:11.038 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:11.039 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:11.040 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.041 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:11.042 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:11.042 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:11.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:11.042 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:10Z" 2023-09-29T22:32:11.042 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:11.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:10Z" } 2023-09-29T22:32:11.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:11.043 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:11.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:11.043 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-29T22:32:16.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:16.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:16.045 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:16.097 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:16.097 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:16.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:16.129+0000 7fb018224880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:16.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:16.131+0000 7fb018224880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:16.143 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:16.144 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:16.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:16.146 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:16.147 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:16.148 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.149 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:16.150 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:16.150 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:16.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:16.150 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:16Z" 2023-09-29T22:32:16.150 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:16.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:16Z" } 2023-09-29T22:32:16.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:16.152 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:16.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:16.152 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:32:21.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:21.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:21.153 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:21.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:21.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:21.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:21.234+0000 7fbfc7612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:21.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:21.235+0000 7fbfc7612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:21.248 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:21.249 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:21.250 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:21.250 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:21.250 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:21.250 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:21.250 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:21.251 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:21.251 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:21.251 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:21.251 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:21.251 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:21.252 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.252 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:21.252 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:21.252 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:21.253 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:21.253 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:21.253 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:21.253 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:21.253 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:21.254 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:21.254 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:21.254 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:21.254 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.255 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:21.255 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:21.255 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:21.255 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:21.255 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.903428_202.1", 2023-09-29T22:32:21.256 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:21.256 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:21.256 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:21.256 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-29T22:32:21.256 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:21.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:21.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:21.257 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:21.258 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:21.258 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:21.258 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:21.258 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:21.258 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:21.259 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:21.259 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:21.259 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:21.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:21.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:21.260 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:21.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.260 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:21.260 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:21.260 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:21.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:21.261 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:21Z" 2023-09-29T22:32:21.261 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:21.263 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.903428_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.903428Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:21Z" } 2023-09-29T22:32:21.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:21.263 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:21.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:21.264 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-29T22:32:26.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:26.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:26.266 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:26.331 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:26.331 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:26.360+0000 7fe0e8b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:26.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:26.360+0000 7fe0e8b26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:26.373 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:26.373 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:26.373 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:26.374 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.375 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:26.376 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:26.377 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:26.378 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.379 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:26.380 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:26.380 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:26.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:26.381 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:26Z" 2023-09-29T22:32:26.381 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:26.382 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:26Z" } 2023-09-29T22:32:26.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:26.382 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:26.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:26.382 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:32:31.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:31.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:31.385 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:31.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:31.436 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:31.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:31.464+0000 7f4239644880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:31.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:31.465+0000 7f4239644880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:31.478 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:31.478 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:31.478 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:31.478 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:31.478 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:31.479 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:31.480 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:31.481 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:32:31.482 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:31.484 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:31.485 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:31Z" 2023-09-29T22:32:31.486 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:31.487 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:31Z" } 2023-09-29T22:32:31.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:31.488 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:31.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:31.488 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:32:36.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:36.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:36.489 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:36.542 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:36.542 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:36.573 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:36.572+0000 7f15a9c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:36.574 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:36.573+0000 7f15a9c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:36.587 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.593 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:36.594 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:36.594 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:36.594 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:36.594 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:36Z" 2023-09-29T22:32:36.594 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:36.595 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:36Z" } 2023-09-29T22:32:36.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:36.595 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:36.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:36.596 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:32:41.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:41.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:41.597 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:41.646 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:41.646 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:41.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:41.672+0000 7f7dcca9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:41.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:41.673+0000 7f7dcca9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:41.684 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:41.684 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:41.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:41.686 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:41.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:41.688 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:41.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:41.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:41.690 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:41.690 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:41.690 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:41.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:32:41.691 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:41.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:41.691 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:41.691 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:32:41.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:41.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:41.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:41.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:41.693 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:41.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:41.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:41.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:41.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:41.694 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:41.694 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:41.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:41.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:41.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:41.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:41.696 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:41Z" 2023-09-29T22:32:41.696 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:41.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:41Z" } 2023-09-29T22:32:41.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:41.697 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:41.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:41.698 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:32:46.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:46.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:46.699 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:46.751 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:46.751 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:46.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:46.783+0000 7ff32fb1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:46.784 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:46.784+0000 7ff32fb1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:46.797 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:46.797 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:46.797 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:46.797 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:46.797 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:46.798 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:46.799 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:46.800 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:46.802 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:46Z" 2023-09-29T22:32:46.803 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:46.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:46Z" } 2023-09-29T22:32:46.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:46.805 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:46.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:46.805 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-29T22:32:51.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:51.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:51.806 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:51.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:51.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:51.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:51.887+0000 7f556fd36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:51.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:51.888+0000 7f556fd36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:51.900 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:51.901 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:51.902 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:51.903 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.980766_211.1", 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:51.904 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:51.905 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:51Z" 2023-09-29T22:32:51.907 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:51.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.980766_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.980766Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:51Z" } 2023-09-29T22:32:51.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:51.908 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:51.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:51.908 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.980766_211.1 2023-09-29T22:32:56.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:56.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:32:56.910 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:32:56.957 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:32:56.957 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:32:56.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:56.986+0000 7ff8df326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:56.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:32:56.986+0000 7ff8df326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:56.999 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:57.000 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:57.001 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:32:57.002 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:32:57.003 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:32:57.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:32:56Z" 2023-09-29T22:32:57.005 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:32:57.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:32:56Z" } 2023-09-29T22:32:57.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:32:57.009 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:57.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:32:57.009 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-29T22:33:02.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:02.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:02.010 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:02.057 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:02.057 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:02.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:02.085+0000 7fbe6c94b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:02.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:02.086+0000 7fbe6c94b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:02.098 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.099 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:02.100 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:02.101 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:02.102 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:33:02.103 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:02.104 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:02.105 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:02.105 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:02.105 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:02.105 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:02.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:02Z" 2023-09-29T22:33:02.106 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:02.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:02Z" } 2023-09-29T22:33:02.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:02.108 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:02.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:02.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-29T22:33:07.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:07.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:07.110 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:07.187 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:07.188 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:07.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:07.217+0000 7ffb5f030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:07.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:07.218+0000 7ffb5f030880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:07.231 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:07.232 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:07.233 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:07.234 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:07.236 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:07Z" 2023-09-29T22:33:07.237 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:07.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:07Z" } 2023-09-29T22:33:07.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:07.239 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:07.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:07.239 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-29T22:33:12.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:12.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:12.240 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:12.289 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:12.289 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:12.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:12.317+0000 7f9eafe26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:12.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:12.318+0000 7f9eafe26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:12.330 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:12.331 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:12.332 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:12.333 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:12.334 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:12.335 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:12Z" 2023-09-29T22:33:12.336 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:12.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:12Z" } 2023-09-29T22:33:12.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:12.337 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:12.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:12.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-29T22:33:17.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2023-09-29T22:33:17.343 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-39 2023-09-29T22:33:17.343 DEBUG:boto:path=/dmuhsu-39/ 2023-09-29T22:33:17.343 DEBUG:boto:auth_path=/dmuhsu-39/ 2023-09-29T22:33:17.343 DEBUG:boto:Method: PUT 2023-09-29T22:33:17.343 DEBUG:boto:Path: /dmuhsu-39/ 2023-09-29T22:33:17.343 DEBUG:boto:Data: 2023-09-29T22:33:17.344 DEBUG:boto:Headers: {} 2023-09-29T22:33:17.344 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:33:17.344 DEBUG:boto:Port: 8000 2023-09-29T22:33:17.344 DEBUG:boto:Params: {} 2023-09-29T22:33:17.344 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:33:17.344 DEBUG:boto:Token: None 2023-09-29T22:33:17.345 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:33:17 GMT /dmuhsu-39/ 2023-09-29T22:33:17.345 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:V81CD62pIBU2hNpYIKCRA1nTdOY= 2023-09-29T22:33:17.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:V81CD62pIBU2hNpYIKCRA1nTdOY=', 'Content-Length': '0'} 2023-09-29T22:33:17.363 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009a4cdb61ee7cfa24-00651750ad-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:17.363 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-40 2023-09-29T22:33:17.363 DEBUG:boto:path=/dmuhsu-40/ 2023-09-29T22:33:17.363 DEBUG:boto:auth_path=/dmuhsu-40/ 2023-09-29T22:33:17.363 DEBUG:boto:Method: PUT 2023-09-29T22:33:17.363 DEBUG:boto:Path: /dmuhsu-40/ 2023-09-29T22:33:17.363 DEBUG:boto:Data: 2023-09-29T22:33:17.364 DEBUG:boto:Headers: {} 2023-09-29T22:33:17.364 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:33:17.364 DEBUG:boto:Port: 8000 2023-09-29T22:33:17.364 DEBUG:boto:Params: {} 2023-09-29T22:33:17.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:33:17.364 DEBUG:boto:Token: None 2023-09-29T22:33:17.364 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:33:17 GMT /dmuhsu-40/ 2023-09-29T22:33:17.365 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BFvsyt+DnI/yh4oCKZP+Fj391zs= 2023-09-29T22:33:17.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BFvsyt+DnI/yh4oCKZP+Fj391zs=', 'Content-Length': '0'} 2023-09-29T22:33:17.398 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000274f94e096309603-00651750ad-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:17.398 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-41 2023-09-29T22:33:17.399 DEBUG:boto:path=/dmuhsu-41/ 2023-09-29T22:33:17.399 DEBUG:boto:auth_path=/dmuhsu-41/ 2023-09-29T22:33:17.399 DEBUG:boto:Method: PUT 2023-09-29T22:33:17.399 DEBUG:boto:Path: /dmuhsu-41/ 2023-09-29T22:33:17.399 DEBUG:boto:Data: 2023-09-29T22:33:17.399 DEBUG:boto:Headers: {} 2023-09-29T22:33:17.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:33:17.399 DEBUG:boto:Port: 8001 2023-09-29T22:33:17.399 DEBUG:boto:Params: {} 2023-09-29T22:33:17.400 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:33:17.400 DEBUG:boto:Token: None 2023-09-29T22:33:17.400 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:33:17 GMT /dmuhsu-41/ 2023-09-29T22:33:17.400 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QJ4NmV5ryMFRkFPlrP4zuJGs1KM= 2023-09-29T22:33:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QJ4NmV5ryMFRkFPlrP4zuJGs1KM=', 'Content-Length': '0'} 2023-09-29T22:33:17.428 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6729d2a4391aaa7-00651750ad-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:17.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.429 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:17.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:17.478 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:17.508 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026797.418167_275.1", 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:33:17.418167Z" 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:17.521 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:33:17.522 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.522 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:33:17.523 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:17.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:33:17.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.523 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:17.574 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:17.574 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:17.607 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:33:17.607+0000 7f0f32526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:17.608 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:33:17.608+0000 7f0f32526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:17.620 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:33:17.621 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:33:17.622 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696026797.418167_275.1", 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:33:17.418167Z", 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:33:17.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:33:17.625 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:33:17.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:33:17.627 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:33:17Z" 2023-09-29T22:33:17.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:33:17.628 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026797.418167_275.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:17.418167Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:17Z" } 2023-09-29T22:33:17.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026797.418167_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:17.418167Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:17.628 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:17.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:17.628 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:33:17.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:33:17.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:17.629 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:17.682 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:17.683 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:17.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:17.711+0000 7f28afe8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:17.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:17.712+0000 7f28afe8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:17.724 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:17.725 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:17.726 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.727 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:17.728 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:17.729 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:17.730 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:17.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:17.731 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:17Z" 2023-09-29T22:33:17.731 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:17.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:17Z" } 2023-09-29T22:33:17.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:17.732 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:17.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:17.732 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:33:22.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:22.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:22.734 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:22.782 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:22.783 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:22.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:22.811+0000 7f4223b31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:22.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:22.812+0000 7f4223b31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:22.825 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:22.826 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:22.827 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:22.828 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:22.829 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:22.830 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:22Z" 2023-09-29T22:33:22.831 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:22.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:22Z" } 2023-09-29T22:33:22.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:22.833 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:22.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:22.833 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-29T22:33:27.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:27.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:27.834 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:27.887 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:27.887 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:27.918+0000 7fe2fc932880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:27.919 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:27.919+0000 7fe2fc932880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:27.932 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:27.932 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:27.933 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:27.934 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:27.935 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:27.936 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:33:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:27.937 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:27.939 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:27.940 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:27.941 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:27Z" 2023-09-29T22:33:27.941 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:27.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:27Z" } 2023-09-29T22:33:27.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:27.944 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:27.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:27.944 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:33:32.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:32.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:32.945 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:32.997 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:32.998 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:33.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:33.028+0000 7f9635d22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:33.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:33.028+0000 7f9635d22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:33.041 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:33.041 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:33.041 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:33.041 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:33.042 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:33.043 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.044 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:33.045 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:33.046 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:33Z" 2023-09-29T22:33:33.047 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:33.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:33Z" } 2023-09-29T22:33:33.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:33.049 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:33.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:33.049 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026483.079039_218.1 2023-09-29T22:33:38.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:38.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:38.051 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:38.099 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:38.099 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:38.126+0000 7f238f612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:38.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:38.127+0000 7f238f612880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:38.139 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.140 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:38.141 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:38.142 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:38.143 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:38.144 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:38.145 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:38Z" 2023-09-29T22:33:38.146 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:38.148 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:38Z" } 2023-09-29T22:33:38.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:38.148 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:38.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:38.149 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:33:43.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:43.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:43.150 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:43.197 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:43.198 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:43.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:43.225+0000 7f30a0d1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:43.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:43.226+0000 7f30a0d1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:33:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:43Z" 2023-09-29T22:33:43.244 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:43.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:43Z" } 2023-09-29T22:33:43.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:43.245 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:43.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:43.246 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-29T22:33:48.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:48.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:48.247 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:48.317 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:48.318 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:48.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:48.348+0000 7f324e525880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:48.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:48.349+0000 7f324e525880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:48.362 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:48.362 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:48.362 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:48.362 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:48.363 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:48.364 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:48.365 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:48.366 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:48.367 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:48.368 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:48.369 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:48Z" 2023-09-29T22:33:48.369 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:48.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:48Z" } 2023-09-29T22:33:48.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:48.370 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:48.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:48.370 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.978724_210.1 2023-09-29T22:33:53.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:53.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:53.372 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:53.422 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:53.422 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:53.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:53.450+0000 7fc845b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:53.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:53.451+0000 7fc845b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:53.463 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:53.463 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:53.463 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:53.464 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:53.466 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:53.467 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:53.468 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:53.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:53.470 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:53Z" 2023-09-29T22:33:53.470 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:53.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:53Z" } 2023-09-29T22:33:53.471 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:53.471 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:53.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:53.471 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:33:58.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:58.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:58.473 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:58.529 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:58.530 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:58.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:58.559+0000 7f0dd97dc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:58.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:33:58.560+0000 7f0dd97dc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:58.584 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:33:58.585 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:58.586 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:58.587 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:58.588 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:33:58.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:33:58.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:33:58.591 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:33:58.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:33:58.591 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:33:58Z" 2023-09-29T22:33:58.592 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:33:58.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:58Z" } 2023-09-29T22:33:58.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:33:58.594 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:58.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:33:58.594 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:34:03.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:03.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:03.710 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:03.755 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:03.755 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:03.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:03.805+0000 7fa629b2a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:03.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:03.806+0000 7fa629b2a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:03.818 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.842044_199.1", 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:03.822 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:03.823 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:03.824 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:03.825 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:03.825 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:03Z" 2023-09-29T22:34:03.825 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:03.826 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.842044_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.842044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:03Z" } 2023-09-29T22:34:03.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:03.826 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:03.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:03.826 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.842044_199.1 2023-09-29T22:34:08.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:08.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:08.828 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:08.875 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:08.875 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:08.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:08.921+0000 7f31fb815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:08.921 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:08.922+0000 7f31fb815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:08.943 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:08.944 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:08.945 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:08.946 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:08.947 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:08.948 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:08Z" 2023-09-29T22:34:08.949 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:08.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:08Z" } 2023-09-29T22:34:08.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:08.951 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:08.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:08.951 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.965243_207.1 2023-09-29T22:34:13.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:13.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:13.952 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:14.000 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:14.000 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:14.043 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:14.043+0000 7feec7c12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:14.044 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:14.044+0000 7feec7c12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:14.060 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:14.060 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:14.061 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.062 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:14.063 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.842044_199.1", 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-29T22:34:14.064 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:14.065 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:14.066 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:14Z" 2023-09-29T22:34:14.067 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:14.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.842044_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.842044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:14Z" } 2023-09-29T22:34:14.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:14.068 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:14.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:14.068 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.842044_199.1 2023-09-29T22:34:19.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:19.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:19.070 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:19.139 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:19.140 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:19.174 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:19.174+0000 7fad83727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:19.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:19.175+0000 7fad83727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:19.188 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:19.188 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.189 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:19.190 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:19.192 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:19.193 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:19.194 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:19.195 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:19.196 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:19Z" 2023-09-29T22:34:19.196 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:19.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:19Z" } 2023-09-29T22:34:19.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:19.197 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:19.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:19.197 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:34:24.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:24.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:24.199 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:24.244 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:24.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:24.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:24.274+0000 7f9849c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:24.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:24.275+0000 7f9849c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:24.288 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:24.289 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:24.290 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:24.291 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:24.292 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:24.293 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:24Z" 2023-09-29T22:34:24.294 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:24.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:24Z" } 2023-09-29T22:34:24.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:24.295 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:24.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:24.296 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.851988_201.1 2023-09-29T22:34:29.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:29.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:29.297 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:29.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:29.371+0000 7f5280830880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:29.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:29.372+0000 7f5280830880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:29.389 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:29.389 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:29.389 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:29.390 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:29.391 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:29.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.825442_197.1", 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-29T22:34:29.393 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:29.394 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:29.395 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:29Z" 2023-09-29T22:34:29.396 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:29.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.825442_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.825442Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:29Z" } 2023-09-29T22:34:29.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:29.397 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:29.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:29.397 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.825442_197.1 2023-09-29T22:34:34.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:34.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:34.398 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:34.451 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:34.451 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:34.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:34.488+0000 7f698b69d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:34.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:34.489+0000 7f698b69d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:34.506 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:34.506 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:34.507 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:34.508 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:34.509 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:34.510 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:34.511 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:34.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:34.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:34:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:34:34.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:34.515 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:34.516 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:34.517 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:34.518 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:34Z" 2023-09-29T22:34:34.518 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:34.519 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:34Z" } 2023-09-29T22:34:34.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:34.519 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:34.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:34.520 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-29T22:34:39.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:39.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:39.521 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:39.599 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:39.599 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:39.673 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:39.672+0000 7fc99c74b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:39.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:39.674+0000 7fc99c74b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:39.711 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:39.711 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:39.711 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:39.711 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:39.712 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:39.713 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:39.715 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:39.716 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:39Z" 2023-09-29T22:34:39.717 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:39.718 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:39Z" } 2023-09-29T22:34:39.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:39.719 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:39.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:39.719 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-29T22:34:44.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:44.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:44.721 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:44.768 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:44.768 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:44.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:44.799+0000 7f2a92126880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:44.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:44.800+0000 7f2a92126880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:44.812 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:44.812 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:44.812 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:44.812 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:44.813 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:44.814 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.815 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:44.816 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:44.817 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:44Z" 2023-09-29T22:34:44.818 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:44.819 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:44Z" } 2023-09-29T22:34:44.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:44.820 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:44.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:44.820 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-29T22:34:49.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:49.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:49.822 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:49.873 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:49.905 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:49.906+0000 7f391e015880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:49.906 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:49.907+0000 7f391e015880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:49.919 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.920 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:49.921 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:49.922 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:49.923 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:49.924 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:49.925 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:49.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:49.926 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:49Z" 2023-09-29T22:34:49.926 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:49.927 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:49Z" } 2023-09-29T22:34:49.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:49.927 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:49.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:49.927 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-29T22:34:54.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:54.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:54.929 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:54.986 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:54.986 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:55.033 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:55.033+0000 7fb1a2811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:55.034 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:34:55.034+0000 7fb1a2811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:55.047 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:55.048 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:55.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:34:55.050 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:55.051 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:34:55.052 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:55.053 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:34:55Z" 2023-09-29T22:34:55.054 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:34:55.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:55Z" } 2023-09-29T22:34:55.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:34:55.056 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:55.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:34:55.056 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:35:00.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:00.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:00.057 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:00.111 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:00.111 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:00.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:00.142+0000 7f84ca812880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:00.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:00.143+0000 7f84ca812880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:00.155 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:00.161 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:00Z" 2023-09-29T22:35:00.162 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:00.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:00Z" } 2023-09-29T22:35:00.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:00.164 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:00.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:00.164 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:35:05.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:05.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:05.165 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:05.213 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:05.213 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:05.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:05.248+0000 7fab3a711880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:05.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:05.249+0000 7fab3a711880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:05.261 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:05Z" 2023-09-29T22:35:05.268 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:05.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:05Z" } 2023-09-29T22:35:05.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:05.270 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:05.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:05.270 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-29T22:35:10.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:10.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:10.272 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:10.322 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:10.322 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:10.363 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:10.363+0000 7f182934b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:10.364 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:10.364+0000 7f182934b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:10.380 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:10.380 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:10.380 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:10.381 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:10Z" 2023-09-29T22:35:10.386 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:10.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:10Z" } 2023-09-29T22:35:10.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:10.388 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:10.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:10.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-29T22:35:15.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:15.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:15.389 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:15.456 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:15.456 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:15.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:15.554+0000 7f2b82a8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:15.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:15.555+0000 7f2b82a8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:15.578 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:15.579 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:15.580 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:15.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:15.582 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:15.583 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:15.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:15Z" 2023-09-29T22:35:15.585 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:15.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:15Z" } 2023-09-29T22:35:15.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:15.586 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:15.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:15.587 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:35:20.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:20.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:20.588 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:20.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:20.634 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:20.664 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:20.664+0000 7f7a7ac22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:20.665 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:20.665+0000 7f7a7ac22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:20.677 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:20.677 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:20.677 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:20.678 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:20.679 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:20.680 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:20.681 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.682 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:20.683 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:20Z" 2023-09-29T22:35:20.684 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:20.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:20Z" } 2023-09-29T22:35:20.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:20.686 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:20.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:20.686 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:35:25.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:25.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:25.688 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:25.733 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:25.734 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:25.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:25.776+0000 7f097cbaf880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:25.778 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:25.777+0000 7f097cbaf880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:25.794 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:25.794 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:25.794 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:25.794 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:25.794 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:25.795 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:25.796 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:25.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:25.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:25.799 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:25.800 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:25Z" 2023-09-29T22:35:25.801 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:25.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:25Z" } 2023-09-29T22:35:25.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:25.802 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:25.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:25.802 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-29T22:35:30.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:30.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:30.804 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:30.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:30.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:30.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:30.907+0000 7f1efbb20880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:30.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:30.908+0000 7f1efbb20880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:30.934 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:30.934 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:30.934 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:30.934 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:30.935 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:30.936 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:30.937 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:30.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:30.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:30Z" 2023-09-29T22:35:30.940 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:30.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:30Z" } 2023-09-29T22:35:30.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:30.942 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:30.942 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:30.942 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-29T22:35:35.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:35.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:35.943 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:35.989 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:35.989 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:36.039 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:36.038+0000 7ff38cf4b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:36.040 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:36.039+0000 7ff38cf4b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:36.063 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:36.063 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:36.063 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:36.063 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:36.064 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:36.065 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.066 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.825442_197.1", 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:36.067 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:36.068 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:36.069 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:36Z" 2023-09-29T22:35:36.070 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:36.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.825442_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.825442Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:36Z" } 2023-09-29T22:35:36.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:36.071 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:36.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:36.071 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.825442_197.1 2023-09-29T22:35:41.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:41.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:41.073 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:41.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:41.151+0000 7f1ed831e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:41.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:41.152+0000 7f1ed831e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:41.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:41.166 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:41.167 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:41.168 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:41.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:41.170 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:41Z" 2023-09-29T22:35:41.171 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:41.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:41Z" } 2023-09-29T22:35:41.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:41.172 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:41.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:41.173 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:35:46.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:46.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:46.174 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:46.224 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:46.225 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:46.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:46.266+0000 7f6139d24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:46.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:46.267+0000 7f6139d24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:46.283 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:46.283 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:46.284 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:46.285 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:46.286 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:46.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:46Z" 2023-09-29T22:35:46.289 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:46.290 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:46Z" } 2023-09-29T22:35:46.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:46.291 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:46.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:46.291 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-29T22:35:51.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:51.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:51.353 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:51.406 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:51.407 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:51.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:51.450+0000 7f6ccbd29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:51.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:51.451+0000 7f6ccbd29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:51.468 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:51.468 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:51.468 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:51.468 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:51.469 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:51.470 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.471 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.962427_205.1", 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:51.472 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:51.473 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:51Z" 2023-09-29T22:35:51.474 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:51.475 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.962427_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.962427Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:51Z" } 2023-09-29T22:35:51.475 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:51.476 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:51.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:51.476 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.962427_205.1 2023-09-29T22:35:56.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:56.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:56.478 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:56.523 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:56.524 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:56.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:56.562+0000 7f6705c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:56.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:35:56.563+0000 7f6705c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:56.575 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:35:56.575 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:56.576 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:56.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:56.578 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:56.579 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:35:56.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:35:56.581 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:35:56.582 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:35:56.582 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:35:56Z" 2023-09-29T22:35:56.582 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:35:56.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:56Z" } 2023-09-29T22:35:56.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:35:56.583 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:56.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:35:56.583 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:36:01.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:01.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:01.585 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:01.630 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:01.630 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:01.677 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:01.677+0000 7f4636922880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:01.678 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:01.678+0000 7f4636922880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:01.690 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:01.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:01.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:01.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:01.694 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:01.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:01.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:01.697 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:01Z" 2023-09-29T22:36:01.697 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:01.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:01Z" } 2023-09-29T22:36:01.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:01.698 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:01.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:01.698 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:36:06.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:06.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:06.700 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:06.748 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:06.748 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:06.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:06.779+0000 7fbd12b23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:06.780 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:06.780+0000 7fbd12b23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.793 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:06.794 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:06.795 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:06.796 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:06.797 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.798 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:06.799 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:06.799 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:06.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:06.799 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:06Z" 2023-09-29T22:36:06.799 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:06.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:06Z" } 2023-09-29T22:36:06.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:06.800 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:06.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:06.800 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-29T22:36:11.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:11.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:11.962 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:12.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:12.234 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:12.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:12.042+0000 7f4772015880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:12.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:12.043+0000 7f4772015880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:12.236 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:12.236 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:12.236 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:12.236 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:12.236 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:12.237 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:12.238 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:12.240 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:12.241 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:12.242 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:12.243 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:12.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:12.243 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:12Z" 2023-09-29T22:36:12.243 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:12.243 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:12Z" } 2023-09-29T22:36:12.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:12.243 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:12.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:12.244 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-29T22:36:17.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:17.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:17.245 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:17.292 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:17.292 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:17.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:17.327+0000 7fa2e2810880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:17.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:17.328+0000 7fa2e2810880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:17.340 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:17.341 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:17.342 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:17.343 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:17.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:17.345 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:17Z" 2023-09-29T22:36:17.346 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:17.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:17Z" } 2023-09-29T22:36:17.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:17.347 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:17.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:17.348 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:36:22.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:22.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:22.349 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:22.395 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:22.396 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:22.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:22.439+0000 7fc36e815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:22.440 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:22.440+0000 7fc36e815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:22.456 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:22.456 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:22.457 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:22.459 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:36:22.460 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:22.461 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:22.462 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:22Z" 2023-09-29T22:36:22.463 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:22.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:22Z" } 2023-09-29T22:36:22.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:22.464 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:22.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:22.464 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-29T22:36:27.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:27.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:27.466 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:27.549 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:27.549 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:27.579+0000 7faa3afa2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:27.582 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:27.580+0000 7faa3afa2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:27.598 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:27.599 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:27.600 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:27.601 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:27.602 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:27.603 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:27Z" 2023-09-29T22:36:27.604 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:27.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:27Z" } 2023-09-29T22:36:27.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:27.605 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:27.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:27.606 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:36:32.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:32.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:32.607 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:32.653 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:32.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:32.706+0000 7f832d325880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:32.861 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:32.707+0000 7f832d325880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:32.863 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:32.863 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:32.863 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:32.863 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:32.864 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:32.864 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:32.864 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:32.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.865 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:32.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:32.865 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:32.865 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:32.865 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:32.866 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:32.866 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:32.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:32.866 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:32.867 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:32.867 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:32.867 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:32.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:32.868 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:32.868 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:32.868 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:32.868 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:32.868 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:32.869 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:32.869 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:32.869 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:32.869 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:32.870 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:32.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:32.870 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:32.871 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:32.871 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:32.871 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:36:32.871 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:32.871 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:32.872 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:32.872 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:36:32.872 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:32.872 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:32.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:32.873 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:32.873 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:32.873 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:32.874 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:32.874 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:32.874 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:32.874 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:32.875 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:32.875 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:32.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:32.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:32.875 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:32.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.876 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:32.876 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:32.876 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:32.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:32.877 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:32Z" 2023-09-29T22:36:32.877 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:32.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:32Z" } 2023-09-29T22:36:32.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:32.878 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:32.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:32.879 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-29T22:36:37.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:37.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:37.880 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:37.929 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:37.929 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:37.970 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:37.969+0000 7fe76dd26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:37.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:37.970+0000 7fe76dd26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:37.986 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:37.986 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:37.986 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:37.986 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:37.986 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:37.987 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:37.988 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:37.989 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:37.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:37.991 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:37Z" 2023-09-29T22:36:37.992 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:37.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:37Z" } 2023-09-29T22:36:37.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:37.994 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:37.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:37.994 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-29T22:36:42.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:42.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:42.995 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:43.075 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:43.076 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:43.154 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:43.155+0000 7f06a2a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:43.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:43.155+0000 7f06a2a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:43.172 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:43.173 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:43.174 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:43.175 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:43.176 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:43.177 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:43.178 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:43.179 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:43Z" 2023-09-29T22:36:43.179 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:43.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:43Z" } 2023-09-29T22:36:43.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:43.180 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:43.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:43.180 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:36:48.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:48.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:48.182 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:48.227 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:48.228 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:48.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:48.282+0000 7fdc30936880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:48.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:48.283+0000 7fdc30936880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:48.297 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:48.298 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:48.299 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:48.300 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:48.302 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:48.303 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:48.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:48.304 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:48Z" 2023-09-29T22:36:48.304 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:48.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:48Z" } 2023-09-29T22:36:48.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:48.305 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:48.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:48.305 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:36:53.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:53.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:53.307 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:53.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:53.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:53.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:53.391+0000 7f24b652f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:53.391 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:53.392+0000 7f24b652f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:53.404 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:53.404 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:53.405 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:53.407 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:53.408 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:53.409 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:53.410 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:53.411 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:53Z" 2023-09-29T22:36:53.411 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:53.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:53Z" } 2023-09-29T22:36:53.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:53.412 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:53.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:53.412 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-29T22:36:58.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:58.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:36:58.414 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:36:58.461 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:58.462 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:58.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:58.501+0000 7fc388fa2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:58.502 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:36:58.502+0000 7fc388fa2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:36:58.515 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:36:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:36:58.515 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:36:58.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:36:58.517 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:58.518 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:58.519 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:36:58.520 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:36:58.521 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:36:58Z" 2023-09-29T22:36:58.522 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:36:58.523 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:36:58Z" } 2023-09-29T22:36:58.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:36:58.523 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:58.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:36:58.524 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:37:03.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:03.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:03.525 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:03.578 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:03.579 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:03.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:03.633+0000 7fcb66529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:03.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:03.634+0000 7fcb66529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:03.653 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:03.654 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:03.655 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:03.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.960282_204.1", 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:03.657 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:03.658 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:03.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:03Z" 2023-09-29T22:37:03.660 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:03.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.960282_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.960282Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:03Z" } 2023-09-29T22:37:03.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:03.662 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:03.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:03.662 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.960282_204.1 2023-09-29T22:37:08.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:08.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:08.663 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:08.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:08.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:08.745 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:08.745+0000 7f4574a2e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:08.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:08.746+0000 7f4574a2e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:08.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:08.760 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:08.762 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:08.763 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:08.764 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:08Z" 2023-09-29T22:37:08.765 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:08.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:08Z" } 2023-09-29T22:37:08.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:08.767 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:08.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:08.767 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:37:13.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:13.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:13.768 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:13.815 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:13.815 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:13.851 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:13.851+0000 7fb6a6c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:13.852 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:13.852+0000 7fb6a6c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:13.864 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:13.865 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:13.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:13.867 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.003827_212.1", 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:13.868 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:13.869 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.870 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:13.871 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:13.871 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:13.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:13.871 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:13Z" 2023-09-29T22:37:13.871 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:13.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.003827_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.003827Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:13Z" } 2023-09-29T22:37:13.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:13.872 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:13.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:13.873 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026483.003827_212.1 2023-09-29T22:37:18.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:18.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:18.874 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:18.927 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:18.928 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:18.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:18.972+0000 7f8a4cf1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:18.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:18.973+0000 7f8a4cf1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:18.992 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:18.993 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:18.994 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:18.995 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:18.996 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:18.997 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:18.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:18.999 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:18Z" 2023-09-29T22:37:18.999 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:19.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:18Z" } 2023-09-29T22:37:19.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:19.000 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:19.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:19.001 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-29T22:37:24.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:24.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:24.002 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:24.055 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:24.055 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:24.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:24.147+0000 7fd46b329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:24.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:24.154+0000 7fd46b329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:24.209 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:24.209 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.210 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:24.211 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:24.212 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:24.213 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:24.214 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:24.215 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:24.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:24Z" 2023-09-29T22:37:24.217 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:24.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:24Z" } 2023-09-29T22:37:24.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:24.219 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:24.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:24.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-29T22:37:29.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:29.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:29.220 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:29.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:29.298+0000 7f756addc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:29.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:29.299+0000 7f756addc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:29.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:29.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:29.317 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:29.318 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:29.319 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:29.320 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:29.321 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:29.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:29.321 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:29Z" 2023-09-29T22:37:29.321 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:29.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:29Z" } 2023-09-29T22:37:29.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:29.322 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:29.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:29.322 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-29T22:37:34.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:34.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:34.324 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:34.369 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:34.369 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:34.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:34.418+0000 7f504cf26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:34.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:34.419+0000 7f504cf26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:34.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:34.443 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:34.444 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:37:34.445 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:34.446 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.447 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:34.448 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:34.448 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:34.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:34.448 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:34Z" 2023-09-29T22:37:34.448 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:34.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:34Z" } 2023-09-29T22:37:34.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:34.449 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:34.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:34.449 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:37:39.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:39.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:39.451 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:39.498 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:39.499 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:39.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:39.540+0000 7fa821126880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:39.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:39.541+0000 7fa821126880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:39.555 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:37:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:39.561 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:39.562 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:39.563 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:39.563 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:39Z" 2023-09-29T22:37:39.563 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:39.564 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:39Z" } 2023-09-29T22:37:39.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:39.564 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:39.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:39.565 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-29T22:37:44.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:44.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:44.566 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:44.632 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:44.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:44.695+0000 7f79fd529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:44.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:44.696+0000 7f79fd529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:44.708 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:44.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:44.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:44.711 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:44.712 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:44.713 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:44.714 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:44.715 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:44.715 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:44.715 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:44Z" 2023-09-29T22:37:44.715 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:44.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:44Z" } 2023-09-29T22:37:44.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:44.716 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:44.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:44.716 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-29T22:37:49.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:49.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:49.718 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:49.763 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:49.764 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:49.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:49.808+0000 7f22b21a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:49.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:49.809+0000 7f22b21a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:49.825 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:49.826 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:49.827 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.828 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:49.829 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:49.830 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:49.831 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:49.832 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:49.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:49.832 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:49Z" 2023-09-29T22:37:49.832 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:49.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:49Z" } 2023-09-29T22:37:49.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:49.833 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:49.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:49.834 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-29T22:37:54.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:54.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:54.835 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:37:54.884 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:37:54.884 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:37:54.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:54.934+0000 7f6ab9abc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:54.934 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:37:54.935+0000 7f6ab9abc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:37:54.951 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:37:54.952 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:54.953 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:54.954 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:37:54.955 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:37:54.956 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:37:54Z" 2023-09-29T22:37:54.957 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:37:54.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:37:54Z" } 2023-09-29T22:37:54.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:37:54.958 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:54.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:37:54.959 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-29T22:37:59.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:59.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:37:59.960 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:00.013 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:00.013 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:00.093 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:00.093+0000 7f6f99210880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:00.097 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:00.097+0000 7f6f99210880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:00.155 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:00.156 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:00.157 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:00.158 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.159 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:00.160 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:00Z" 2023-09-29T22:38:00.160 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:00.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:00Z" } 2023-09-29T22:38:00.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:00.161 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:00.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:00.161 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-29T22:38:05.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:05.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:05.162 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:05.208 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:05.208 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:05.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:05.242+0000 7f344642e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:05.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:05.243+0000 7f344642e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:05.260 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:05.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:05.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:05.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:05.264 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:05.265 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:05.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:05.267 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:05Z" 2023-09-29T22:38:05.267 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:05.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:05Z" } 2023-09-29T22:38:05.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:05.268 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:05.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:05.268 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-29T22:38:10.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:10.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:10.269 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:10.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:10.320 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:10.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:10.361+0000 7f4adef22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:10.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:10.361+0000 7f4adef22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:10.382 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:10.382 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:10.383 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:10.384 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:10.385 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:10.386 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:10.387 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:10.388 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:10.389 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:10.390 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:10.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:10.390 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:10Z" 2023-09-29T22:38:10.390 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:10.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:10Z" } 2023-09-29T22:38:10.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:10.391 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:10.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:10.392 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.978724_210.1 2023-09-29T22:38:15.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:15.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:15.393 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:15.444 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:15.445 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:15.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:15.490+0000 7f15e740a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:15.491 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:15.491+0000 7f15e740a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:15.511 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:15.512 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:15.513 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:15.514 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:15.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:15Z" 2023-09-29T22:38:15.517 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:15.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:15Z" } 2023-09-29T22:38:15.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:15.519 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:15.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:15.519 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-29T22:38:20.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:20.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:20.520 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:20.571 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:20.571 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:20.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:20.626+0000 7f21cafaf880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:20.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:20.631+0000 7f21cafaf880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:20.689 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:20.689 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:20.689 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:20.690 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:20.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:20.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:20.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:20.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:20.696 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:20Z" 2023-09-29T22:38:20.696 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:20.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:20Z" } 2023-09-29T22:38:20.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:20.697 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:20.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:20.697 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-29T22:38:25.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2023-09-29T22:38:25.702 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-42 2023-09-29T22:38:25.702 DEBUG:boto:path=/dmuhsu-42/ 2023-09-29T22:38:25.702 DEBUG:boto:auth_path=/dmuhsu-42/ 2023-09-29T22:38:25.702 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.702 DEBUG:boto:Path: /dmuhsu-42/ 2023-09-29T22:38:25.702 DEBUG:boto:Data: 2023-09-29T22:38:25.703 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.703 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:38:25.703 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.703 DEBUG:boto:Params: {} 2023-09-29T22:38:25.703 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:38:25.703 DEBUG:boto:Token: None 2023-09-29T22:38:25.703 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-42/ 2023-09-29T22:38:25.704 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mS8SwebejWh2hJKXzZd0gRzIxfM= 2023-09-29T22:38:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mS8SwebejWh2hJKXzZd0gRzIxfM=', 'Content-Length': '0'} 2023-09-29T22:38:25.722 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000357eb9b8cfac14a5-00651751e1-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.723 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-43 2023-09-29T22:38:25.723 DEBUG:boto:path=/dmuhsu-43/ 2023-09-29T22:38:25.723 DEBUG:boto:auth_path=/dmuhsu-43/ 2023-09-29T22:38:25.723 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.723 DEBUG:boto:Path: /dmuhsu-43/ 2023-09-29T22:38:25.723 DEBUG:boto:Data: 2023-09-29T22:38:25.723 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.723 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:38:25.723 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.724 DEBUG:boto:Params: {} 2023-09-29T22:38:25.724 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:38:25.724 DEBUG:boto:Token: None 2023-09-29T22:38:25.724 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-43/ 2023-09-29T22:38:25.724 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wTP+3Lu1wTWZLQywbEEmoX3UVEs= 2023-09-29T22:38:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wTP+3Lu1wTWZLQywbEEmoX3UVEs=', 'Content-Length': '0'} 2023-09-29T22:38:25.753 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ecf79f773654612-00651751e1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.754 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-44 2023-09-29T22:38:25.754 DEBUG:boto:path=/dmuhsu-44/ 2023-09-29T22:38:25.754 DEBUG:boto:auth_path=/dmuhsu-44/ 2023-09-29T22:38:25.754 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.754 DEBUG:boto:Path: /dmuhsu-44/ 2023-09-29T22:38:25.754 DEBUG:boto:Data: 2023-09-29T22:38:25.754 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.755 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:38:25.755 DEBUG:boto:Port: 8001 2023-09-29T22:38:25.755 DEBUG:boto:Params: {} 2023-09-29T22:38:25.755 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:38:25.755 DEBUG:boto:Token: None 2023-09-29T22:38:25.755 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-44/ 2023-09-29T22:38:25.755 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DfVTztQPbtQq1dtr4ylj8su/YGM= 2023-09-29T22:38:25.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DfVTztQPbtQq1dtr4ylj8su/YGM=', 'Content-Length': '0'} 2023-09-29T22:38:25.790 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a609a526ee4f69c6-00651751e1-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.790 DEBUG:boto:path=/dmuhsu-42/ 2023-09-29T22:38:25.790 DEBUG:boto:auth_path=/dmuhsu-42/ 2023-09-29T22:38:25.790 DEBUG:boto:Method: HEAD 2023-09-29T22:38:25.790 DEBUG:boto:Path: /dmuhsu-42/ 2023-09-29T22:38:25.791 DEBUG:boto:Data: 2023-09-29T22:38:25.791 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.791 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:38:25.791 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.791 DEBUG:boto:Params: {} 2023-09-29T22:38:25.791 DEBUG:boto:Token: None 2023-09-29T22:38:25.791 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-42/ 2023-09-29T22:38:25.791 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Uwgb6JNU3T/GT8GE/IZxx8XO4jU= 2023-09-29T22:38:25.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Uwgb6JNU3T/GT8GE/IZxx8XO4jU=', 'Content-Length': '0'} 2023-09-29T22:38:25.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002961ea9b915c1131-00651751e1-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.794 DEBUG:boto:path=/dmuhsu-42/key 2023-09-29T22:38:25.794 DEBUG:boto:auth_path=/dmuhsu-42/key 2023-09-29T22:38:25.794 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.794 DEBUG:boto:Path: /dmuhsu-42/key 2023-09-29T22:38:25.794 DEBUG:boto:Data: 2023-09-29T22:38:25.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-29T22:38:25.794 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:38:25.795 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.795 DEBUG:boto:Params: {} 2023-09-29T22:38:25.795 DEBUG:boto:Token: None 2023-09-29T22:38:25.795 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-42/key 2023-09-29T22:38:25.795 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SpONFyfpUrHjJLBk3/TLY3AKHuA= 2023-09-29T22:38:25.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SpONFyfpUrHjJLBk3/TLY3AKHuA='} 2023-09-29T22:38:25.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acf6d2764d0e385c-00651751e1-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.805 DEBUG:boto:path=/dmuhsu-43/ 2023-09-29T22:38:25.805 DEBUG:boto:auth_path=/dmuhsu-43/ 2023-09-29T22:38:25.805 DEBUG:boto:Method: HEAD 2023-09-29T22:38:25.805 DEBUG:boto:Path: /dmuhsu-43/ 2023-09-29T22:38:25.805 DEBUG:boto:Data: 2023-09-29T22:38:25.805 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.805 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:38:25.805 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.805 DEBUG:boto:Params: {} 2023-09-29T22:38:25.806 DEBUG:boto:Token: None 2023-09-29T22:38:25.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-43/ 2023-09-29T22:38:25.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iGftnf76Bk6nFZPgqL90llq6HYg= 2023-09-29T22:38:25.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iGftnf76Bk6nFZPgqL90llq6HYg=', 'Content-Length': '0'} 2023-09-29T22:38:25.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33a30cd95cf0f86-00651751e1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.809 DEBUG:boto:path=/dmuhsu-43/key 2023-09-29T22:38:25.809 DEBUG:boto:auth_path=/dmuhsu-43/key 2023-09-29T22:38:25.809 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.809 DEBUG:boto:Path: /dmuhsu-43/key 2023-09-29T22:38:25.809 DEBUG:boto:Data: 2023-09-29T22:38:25.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-29T22:38:25.809 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:38:25.809 DEBUG:boto:Port: 8000 2023-09-29T22:38:25.810 DEBUG:boto:Params: {} 2023-09-29T22:38:25.810 DEBUG:boto:Token: None 2023-09-29T22:38:25.810 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-43/key 2023-09-29T22:38:25.810 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tXjfugpsiuw1S06lZyOvRy3ZSzk= 2023-09-29T22:38:25.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tXjfugpsiuw1S06lZyOvRy3ZSzk='} 2023-09-29T22:38:25.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa72960c07eb567f-00651751e1-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.818 DEBUG:boto:path=/dmuhsu-44/ 2023-09-29T22:38:25.818 DEBUG:boto:auth_path=/dmuhsu-44/ 2023-09-29T22:38:25.818 DEBUG:boto:Method: HEAD 2023-09-29T22:38:25.818 DEBUG:boto:Path: /dmuhsu-44/ 2023-09-29T22:38:25.818 DEBUG:boto:Data: 2023-09-29T22:38:25.818 DEBUG:boto:Headers: {} 2023-09-29T22:38:25.818 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:38:25.818 DEBUG:boto:Port: 8001 2023-09-29T22:38:25.819 DEBUG:boto:Params: {} 2023-09-29T22:38:25.819 DEBUG:boto:Token: None 2023-09-29T22:38:25.819 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-44/ 2023-09-29T22:38:25.819 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TWMm+xuK6xGnqJRg3ryHlnGCQ/o= 2023-09-29T22:38:25.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TWMm+xuK6xGnqJRg3ryHlnGCQ/o=', 'Content-Length': '0'} 2023-09-29T22:38:25.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000892e8168f3ac7b9b-00651751e1-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.822 DEBUG:boto:path=/dmuhsu-44/key 2023-09-29T22:38:25.822 DEBUG:boto:auth_path=/dmuhsu-44/key 2023-09-29T22:38:25.822 DEBUG:boto:Method: PUT 2023-09-29T22:38:25.822 DEBUG:boto:Path: /dmuhsu-44/key 2023-09-29T22:38:25.822 DEBUG:boto:Data: 2023-09-29T22:38:25.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-29T22:38:25.822 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:38:25.822 DEBUG:boto:Port: 8001 2023-09-29T22:38:25.822 DEBUG:boto:Params: {} 2023-09-29T22:38:25.823 DEBUG:boto:Token: None 2023-09-29T22:38:25.823 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 29 Sep 2023 22:38:25 GMT /dmuhsu-44/key 2023-09-29T22:38:25.823 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AnTTmOhWKQjlvd60w9F9HDzMEII= 2023-09-29T22:38:25.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AnTTmOhWKQjlvd60w9F9HDzMEII='} 2023-09-29T22:38:25.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e1de1675efc7524-00651751e1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:25.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:25.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:25.834 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:25.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:25.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:25.925 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:26.004 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:38:25.772402Z" 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:26.005 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:26.006 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:38:26.006 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:26.006 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:38:26.007 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:26.007 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:38:26.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:26.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:26.007 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:26.059 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:26.059 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:26.092 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:38:26.092+0000 7fc39191e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:26.092 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:38:26.093+0000 7fc39191e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:38:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:38:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.715044_276.1", 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.715044Z", 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:38:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:38:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:38:26Z" 2023-09-29T22:38:26.112 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:38:26.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.715044_276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.715044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:26Z" } 2023-09-29T22:38:26.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.715044_276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.715044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:26.113 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:26.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.715044_276.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:26.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696027105.715044_276.1 2023-09-29T22:38:31.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:31.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:31.115 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:31.168 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:31.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:31.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:38:31.201+0000 7f7a3c813880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:31.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:38:31.203+0000 7f7a3c813880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:31.215 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:38:31.216 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:38:31.216 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:38:31.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:38:31.216 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:38:31.216 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:31.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:31.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.217 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:38:31.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:31.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:38:31.218 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:31.218 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:31.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:31.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:38:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:31.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:31.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:31.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:38:31.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:31.221 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:31.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:38:31.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:31.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:31.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:31.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:38:31.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:31.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:31.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:31.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:38:31.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:31.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:31.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:38:31.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:31.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:31.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:31.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:38:31.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:31.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:31.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:38:31.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:38:31.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:38:31.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:38:31.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:31.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:38:31.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:38:31.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:38:31.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.228 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:38:31.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:38:31.228 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:38:31.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:38:31.229 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:38:31Z" 2023-09-29T22:38:31.229 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:38:31.230 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:31Z" } 2023-09-29T22:38:31.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:31.231 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:31.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:31.231 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:38:31.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:38:31.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:31.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:31.232 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:31.277 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:31.277 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:31.321+0000 7f52dd34b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:31.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:31.322+0000 7f52dd34b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:31.340 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:31.341 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:31.342 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.849856_200.1", 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-29T22:38:31.344 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:31.345 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:31.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:31.347 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:31Z" 2023-09-29T22:38:31.347 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:31.348 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.849856_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.849856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:31Z" } 2023-09-29T22:38:31.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:31.348 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:31.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:31.349 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-29T22:38:36.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:36.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:36.350 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:36.398 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:36.398 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:36.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:36.433+0000 7fce04527880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:36.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:36.434+0000 7fce04527880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:36.452 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:36.452 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.453 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:36.454 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:36.455 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:36.456 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:36.457 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:36.458 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:36.459 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:36.460 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:36Z" 2023-09-29T22:38:36.461 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:36.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:36Z" } 2023-09-29T22:38:36.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:36.462 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:36.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:36.462 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-29T22:38:38.706 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:38:38.705+0000 7f303f796640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:38:39.932 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-29T22:38:39.932+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:38:41.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:41.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:41.463 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:41.535 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:41.535 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:41.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:41.629+0000 7fd09b336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:41.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:41.630+0000 7fd09b336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:41.656 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:41.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:41.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:41.659 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:41.660 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:41.661 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:41.662 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:41.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:41.663 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:41Z" 2023-09-29T22:38:41.663 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:41.664 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:41Z" } 2023-09-29T22:38:41.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:41.664 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:41.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:41.664 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-29T22:38:46.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:46.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:46.666 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:46.711 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:46.712 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:46.763+0000 7f0a69467880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:46.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:46.764+0000 7f0a69467880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:46.787 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:46.787 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:46.788 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:46.789 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:46.790 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:46.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:46.792 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:46Z" 2023-09-29T22:38:46.793 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:46.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:46Z" } 2023-09-29T22:38:46.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:46.795 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:46.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:46.795 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-29T22:38:51.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:51.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:51.796 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:51.847 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:51.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:51.884+0000 7f501f330880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:51.884 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:51.885+0000 7f501f330880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:51.905 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:51.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:51.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:51.908 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:51.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:51.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:51.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:51.912 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:51Z" 2023-09-29T22:38:51.912 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:51.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:51Z" } 2023-09-29T22:38:51.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:51.913 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:51.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:51.913 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026483.079039_218.1 2023-09-29T22:38:56.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:56.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:38:56.915 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:38:56.963 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:56.963 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:57.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:57.006+0000 7fa116e8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:57.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:38:57.007+0000 7fa116e8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:38:57.021 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:38:57.021 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:38:57.021 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:38:57.021 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:38:57.021 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:57.022 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:38:57.023 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:57.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:38:57.026 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:38:57.027 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:38:57.029 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.030 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:38:57.031 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:38:57.031 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:38:57.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:38:57.031 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:38:57Z" 2023-09-29T22:38:57.031 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:38:57.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:38:57Z" } 2023-09-29T22:38:57.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:38:57.034 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:57.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:38:57.034 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-29T22:39:02.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:02.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:02.036 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:02.082 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:02.082 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:02.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:02.125+0000 7f2437727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:02.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:02.125+0000 7f2437727880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:02.141 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:02.142 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:02.143 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:02.144 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:02.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:02.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:02.147 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:02.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:02.147 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:02Z" 2023-09-29T22:39:02.147 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:02.148 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:02Z" } 2023-09-29T22:39:02.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:02.148 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:02.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:02.148 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-29T22:39:07.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:07.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:07.150 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:07.197 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:07.198 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:07.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:07.241+0000 7fedacb26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:07.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:07.242+0000 7fedacb26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.259 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:07.260 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:07.261 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:39:07.262 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:07.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.264 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:07.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:07.265 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:07.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:07.265 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:07Z" 2023-09-29T22:39:07.265 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:07.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:07Z" } 2023-09-29T22:39:07.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:07.266 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:07.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:07.266 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.799906_193.1 2023-09-29T22:39:12.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:12.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:12.268 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:12.325 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:12.325 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:12.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:12.365+0000 7f7bafa13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:12.366 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:12.366+0000 7f7bafa13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:12.379 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:12.379 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:12.379 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:12.379 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:12.380 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:12.381 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:12.382 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:12.383 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:12.384 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:12.385 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.386 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:12.386 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:12.386 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:12.386 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:12.386 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:12.387 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:12.388 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:12Z" 2023-09-29T22:39:12.389 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:12.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:12Z" } 2023-09-29T22:39:12.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:12.391 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:12.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:12.391 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-29T22:39:17.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:17.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:17.392 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:17.457 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:17.457 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:17.513 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:17.513+0000 7ff3a6b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:17.514 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:17.514+0000 7ff3a6b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:17.530 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:17.530 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:17.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:17.532 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:17.533 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:17.534 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:17.535 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:17.536 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:17.537 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:17.538 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:17.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:17.538 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:17Z" 2023-09-29T22:39:17.538 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:17.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:17Z" } 2023-09-29T22:39:17.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:17.539 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:17.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:17.540 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026483.079039_218.1 2023-09-29T22:39:22.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:22.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:22.542 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:22.587 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:22.587 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:22.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:22.637+0000 7f8995226880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:22.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:22.638+0000 7f8995226880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:22.653 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:22.654 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:22.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:22.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:22.657 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:22.658 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:22.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:22.660 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:22Z" 2023-09-29T22:39:22.660 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:22.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:22Z" } 2023-09-29T22:39:22.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:22.661 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:22.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:22.661 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-29T22:39:27.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:27.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:27.663 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:27.708 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:27.709 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:27.779 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:27.778+0000 7fedd994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:27.783 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:27.782+0000 7fedd994b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:27.829 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:27.829 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:27.829 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:27.830 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:27.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:27.832 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:39:27.833 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:27.834 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:27.835 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:27.835 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:27.835 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:27.835 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:27.835 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.836 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:27.837 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:27.837 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:27.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:27.837 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:27Z" 2023-09-29T22:39:27.837 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:27.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:27Z" } 2023-09-29T22:39:27.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:27.839 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:27.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:27.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:39:32.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:32.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:32.840 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:32.886 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:32.886 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:32.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:32.930+0000 7f4dddf27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:32.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:32.931+0000 7f4dddf27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:32.946 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:32.947 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:32.949 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:39:32.950 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:32.951 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:32.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:32Z" 2023-09-29T22:39:32.953 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:32.954 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:32Z" } 2023-09-29T22:39:32.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:32.955 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:32.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:32.955 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-29T22:39:37.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:37.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:37.956 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:38.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:38.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:38.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:38.064+0000 7f1422815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:38.066 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:38.065+0000 7f1422815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:38.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:38.090 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:38.091 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:38.092 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:38.093 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:38.094 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:38Z" 2023-09-29T22:39:38.095 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:38.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:38Z" } 2023-09-29T22:39:38.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:38.096 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:38.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:38.097 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.978724_210.1 2023-09-29T22:39:43.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:43.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:43.098 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:43.145 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:43.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:43.190+0000 7f67263af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:43.191 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:43.191+0000 7f67263af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:43.209 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:43.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:43.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.963214_206.1", 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:43.213 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:43.214 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:43.215 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:43Z" 2023-09-29T22:39:43.216 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:43.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.963214_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.963214Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:43Z" } 2023-09-29T22:39:43.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.963214_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.963214Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:43.217 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:43.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.963214_206.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:43.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.963214_206.1 2023-09-29T22:39:48.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:48.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:48.219 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:48.293 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:48.294 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:48.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:48.383+0000 7f9626a12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:48.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:48.385+0000 7f9626a12880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:48.401 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:48.402 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:48.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:48.404 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:48.405 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:48.406 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:48.407 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:48.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:48.408 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:48Z" 2023-09-29T22:39:48.408 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:48.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:48Z" } 2023-09-29T22:39:48.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:48.409 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:48.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:48.409 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-29T22:39:53.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:53.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:53.411 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:53.479 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:53.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:53.525+0000 7fc9de810880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:53.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:53.526+0000 7fc9de810880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:53.544 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:53.545 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:53.546 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:53.547 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:53Z" 2023-09-29T22:39:53.548 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:53.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:53Z" } 2023-09-29T22:39:53.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:53.550 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:53.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:53.550 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:39:58.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:58.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:39:58.552 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:39:58.598 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:39:58.598 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:39:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:58.636+0000 7f828f210880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:58.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:39:58.636+0000 7f828f210880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:39:58.652 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:39:58.653 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:58.654 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:58.655 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:39:58.656 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:39:58.657 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:39:58Z" 2023-09-29T22:39:58.658 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:39:58.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:39:58Z" } 2023-09-29T22:39:58.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:39:58.659 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:58.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:39:58.660 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-29T22:40:03.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:03.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:03.661 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:03.732 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:03.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:03.799+0000 7f30d0644880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:03.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:03.800+0000 7f30d0644880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:03.813 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:03.813 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:03.814 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:03.815 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:03.816 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:03.817 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.849856_200.1", 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:03.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:03.819 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:03.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:03Z" 2023-09-29T22:40:03.821 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:03.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.849856_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.849856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:03Z" } 2023-09-29T22:40:03.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:03.823 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:03.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:03.823 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-29T22:40:08.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:08.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:08.825 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:08.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:08.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:08.993+0000 7fb7b6734880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:08.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:08.994+0000 7fb7b6734880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:09.037 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:09.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:09.039 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:09.040 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:09.041 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:09.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:09.043 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:09.044 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:08Z" 2023-09-29T22:40:09.045 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:09.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:08Z" } 2023-09-29T22:40:09.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:09.046 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:09.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:09.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-29T22:40:14.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:14.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:14.048 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:14.096 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:14.097 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:14.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:14.131+0000 7f6d42226880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:14.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:14.132+0000 7f6d42226880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:14.147 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:14.147 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:14.147 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:14.148 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:14.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:14.150 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.799906_193.1", 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:14.151 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:14.152 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.153 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:14.154 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:14.154 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:14.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:14.154 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:14Z" 2023-09-29T22:40:14.154 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:14.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.799906_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.799906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:14Z" } 2023-09-29T22:40:14.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:14.155 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:14.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:14.155 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.799906_193.1 2023-09-29T22:40:19.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:19.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:19.157 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:19.219 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:19.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:19.252+0000 7fd53df36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:19.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:19.253+0000 7fd53df36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:19.271 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:19.271 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:19.272 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:19.273 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.903428_202.1", 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:19.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:19.276 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:19Z" 2023-09-29T22:40:19.277 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:19.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.903428_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.903428Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:19Z" } 2023-09-29T22:40:19.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:19.279 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:19.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:19.279 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.903428_202.1 2023-09-29T22:40:24.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:24.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:24.280 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:24.328 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:24.328 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:24.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:24.371+0000 7fcddb419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:24.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:24.372+0000 7fcddb419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:24.392 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:24.392 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:24.392 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:24.392 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:24.392 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:24.393 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:24.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.823352_196.1", 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:24.396 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:24.397 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:24.398 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:24.399 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:24.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:24.399 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:24Z" 2023-09-29T22:40:24.399 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:24.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.823352_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.823352Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:24Z" } 2023-09-29T22:40:24.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:24.400 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:24.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:24.400 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-29T22:40:29.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:29.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:29.402 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:29.476 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:29.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:29.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:29.548+0000 7f03b5e17880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:29.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:29.549+0000 7f03b5e17880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:29.564 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:29.564 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:29.564 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:29.565 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.566 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:29.567 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:29.568 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:29.569 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:29.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:29.571 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:29Z" 2023-09-29T22:40:29.571 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:29.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:29Z" } 2023-09-29T22:40:29.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:29.572 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:29.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:29.572 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:40:34.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:34.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:34.574 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:34.639 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:34.639 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:34.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:34.671+0000 7ff796120880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:34.672 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:34.672+0000 7ff796120880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:34.691 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:34.691 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:34.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:34.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:40:34.695 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:34.696 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.697 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:34.698 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:34.698 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:34.698 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:34.698 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:34Z" 2023-09-29T22:40:34.698 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:34.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:34Z" } 2023-09-29T22:40:34.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:34.699 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:34.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:34.700 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:40:39.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:39.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:39.701 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:39.749 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:39.750 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:39.786 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:39.785+0000 7fc594a10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:39.787 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:39.786+0000 7fc594a10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:39.803 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:39.803 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:39.803 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:39.803 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:39.804 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:39.805 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.806 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.842044_199.1", 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:39.808 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:39Z" 2023-09-29T22:40:39.809 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:39.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.842044_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.842044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:39Z" } 2023-09-29T22:40:39.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:39.811 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:39.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:39.811 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-29T22:40:44.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:44.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:44.812 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:44.860 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:44.860 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:44.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:44.910+0000 7fcad931c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:44.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:44.910+0000 7fcad931c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:44.929 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:45.089 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:45.090 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:45.091 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:45.092 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:45.093 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:45.094 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:45.095 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:45.096 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:45.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:45.096 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:44Z" 2023-09-29T22:40:45.096 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:45.097 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:44Z" } 2023-09-29T22:40:45.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:45.098 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:45.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:45.098 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:40:50.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:50.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:50.099 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:50.182 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:50.182 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:50.264+0000 7fdc54f22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:50.264+0000 7fdc54f22880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:50.281 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:50.281 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:50.281 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:50.281 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:50.281 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:50.282 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:50.283 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:50.284 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.977901_209.1", 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:50.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:50.286 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:50Z" 2023-09-29T22:40:50.287 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:50.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.977901_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.977901Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:50Z" } 2023-09-29T22:40:50.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:50.289 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:50.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:50.289 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.977901_209.1 2023-09-29T22:40:55.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:55.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:40:55.290 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:40:55.336 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:40:55.336 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:40:55.382 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:55.382+0000 7fe7ba267880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:55.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:40:55.383+0000 7fe7ba267880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:40:55.403 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:40:55.403 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:40:55.403 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:40:55.403 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:55.404 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:55.405 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:55.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:40:55.407 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:55.408 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:40:55.409 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:40:55.409 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:40:55.409 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:40:55.409 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:40:55.410 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:40:55.410 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:40:55.410 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:40:55.410 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:40:55.410 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:40:55.411 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:40:55.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:40:55.412 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:40:55Z" 2023-09-29T22:40:55.412 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:40:55.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:40:55Z" } 2023-09-29T22:40:55.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:40:55.414 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:55.414 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:40:55.414 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-29T22:41:00.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:00.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:00.416 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:00.464 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:00.464 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:00.505 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:00.505+0000 7fd01542e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:00.506 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:00.506+0000 7fd01542e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:00.522 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:00.522 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:00.522 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:00.523 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:00.524 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:00.525 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:00.526 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:00.527 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:00.528 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:00.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:00.529 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:00Z" 2023-09-29T22:41:00.529 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:00.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:00Z" } 2023-09-29T22:41:00.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:00.530 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:00.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:00.530 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-29T22:41:05.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:05.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:05.532 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:05.585 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:05.722 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:05.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:05.649+0000 7fb678d1a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:05.723 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:05.651+0000 7fb678d1a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:05.727 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:05.728 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:05.729 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:05.730 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:05.731 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:05.732 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:05.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:05.734 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:05Z" 2023-09-29T22:41:05.734 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:05.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:05Z" } 2023-09-29T22:41:05.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:05.734 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:05.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:05.735 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-29T22:41:10.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:10.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:10.736 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:10.782 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:10.782 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:10.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:10.833+0000 7f0752067880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:10.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:10.834+0000 7f0752067880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:10.852 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:10.853 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:10.853 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:10.853 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:10.853 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:10.854 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:10.854 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:10.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.854 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:10.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:10.855 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:10.857 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:10.857 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:10.857 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:10.857 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:10.857 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:41:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:10Z" 2023-09-29T22:41:10.864 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:10.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:10Z" } 2023-09-29T22:41:10.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:10.865 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:10.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:10.866 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:41:15.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:15.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:15.870 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:15.916 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:15.917 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:15.955 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:15.955+0000 7fd330336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:15.956 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:15.956+0000 7fd330336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:15.970 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:15.971 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:15.972 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:15.973 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:15.974 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:15.975 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:15.975 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:15.975 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:15.975 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:15.975 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:15.976 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:15.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:15.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:15.978 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:15.978 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:15.978 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:15.979 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:15.979 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:15.979 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:15.979 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:15.979 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:15Z" 2023-09-29T22:41:15.981 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:15.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:15Z" } 2023-09-29T22:41:15.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:15.983 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:15.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:15.983 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-29T22:41:20.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:20.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:20.984 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:21.033 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:21.033 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:21.064+0000 7f9ef5b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:21.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:21.065+0000 7f9ef5b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:21.077 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:21.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:21.079 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:21.080 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:21.081 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.796890_191.1", 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.082 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:21.083 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:21Z" 2023-09-29T22:41:21.084 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:21.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.796890_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.796890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:21Z" } 2023-09-29T22:41:21.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:21.086 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:21.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:21.086 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-29T22:41:26.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:26.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:26.088 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:26.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:26.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:26.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:26.172+0000 7f8d44a0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:26.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:26.172+0000 7f8d44a0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:26.799 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:27.425 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:27.425 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:27.425 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:27.426 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:27.427 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:27.428 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:27.428 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:27.428 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:41:27.428 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:27.428 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:27.429 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:27.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:26Z" 2023-09-29T22:41:27.431 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:27.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:26Z" } 2023-09-29T22:41:27.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:27.432 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:27.432 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:27.432 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-29T22:41:32.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:32.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:32.433 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:32.480 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:32.480 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:32.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:32.531+0000 7f95af1a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:32.532 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:32.532+0000 7f95af1a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:32.551 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:32.552 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:32.554 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:32.555 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:32.556 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:32.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:32.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:32.559 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:32Z" 2023-09-29T22:41:32.559 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:32.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:32Z" } 2023-09-29T22:41:32.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:32.560 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:32.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:32.560 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-29T22:41:37.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:37.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:37.562 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:37.610 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:37.611 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:37.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:37.655+0000 7f7dcae0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:37.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:37.655+0000 7f7dcae0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:37.673 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:37.673 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:37.673 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:37.673 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:37.673 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:37.674 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:37.676 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.962427_205.1", 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:37.677 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:37.678 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:37.679 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:37Z" 2023-09-29T22:41:37.680 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:37.681 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.962427_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.962427Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:37Z" } 2023-09-29T22:41:37.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:37.682 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:37.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:37.682 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.962427_205.1 2023-09-29T22:41:42.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:42.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:42.684 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:42.739 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:42.740 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:42.869 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:42.869+0000 7f89ec5a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:42.874 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:42.874+0000 7f89ec5a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:42.906 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:42.906 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:42.906 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:42.907 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:42.907 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:42.907 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:42.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:42.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.908 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:42.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:42.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:42.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:42.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:42.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:42.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:42.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:42.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:42.909 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:42.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:42.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:42.912 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:42.913 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:42.914 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:42Z" 2023-09-29T22:41:42.915 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:42.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:42Z" } 2023-09-29T22:41:42.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:42.917 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:42.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:42.917 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-29T22:41:47.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:47.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:47.919 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:47.964 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:47.965 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:48.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:48.009+0000 7f2fee524880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:48.137 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:48.010+0000 7f2fee524880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:48.138 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:48.138 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:48.139 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:48.140 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.141 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.849856_200.1", 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:48.142 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:48.143 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:48Z" 2023-09-29T22:41:48.144 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:48.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.849856_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.849856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:48Z" } 2023-09-29T22:41:48.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:48.145 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:48.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:48.145 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-29T22:41:53.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:53.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:53.146 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:53.194 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:53.194 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:53.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:53.239+0000 7f96e4932880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:53.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:53.240+0000 7f96e4932880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:53.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:53.267 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:53.268 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:53.269 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.794634_190.1", 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:53.270 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:53.271 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:53Z" 2023-09-29T22:41:53.272 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:53.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.794634_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.794634Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:53Z" } 2023-09-29T22:41:53.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:53.273 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:53.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:53.274 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-29T22:41:58.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:58.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:41:58.275 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:41:58.328 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:41:58.329 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:41:58.381 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:58.381+0000 7f8bd6c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:58.392 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:41:58.392+0000 7f8bd6c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:41:58.465 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:41:58.466 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:58.467 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:58.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:58.469 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:41:58.470 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:41:58.471 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:41:58Z" 2023-09-29T22:41:58.472 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:41:58.474 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:41:58Z" } 2023-09-29T22:41:58.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:41:58.474 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:58.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:41:58.475 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-29T22:42:03.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:03.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:03.477 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:03.526 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:03.526 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:03.579 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:03.579+0000 7f562f230880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:03.580 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:03.580+0000 7f562f230880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:03.604 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:03.605 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:03.606 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:03.607 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.849856_200.1", 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.608 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:03.609 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:03Z" 2023-09-29T22:42:03.610 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:03.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.849856_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.849856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:03Z" } 2023-09-29T22:42:03.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:03.612 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:03.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:03.612 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-29T22:42:08.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:08.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:08.613 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:08.660 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:08.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:08.697+0000 7fe4af72c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:08.936 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:08.698+0000 7fe4af72c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:08.937 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:08.938 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:08.939 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:08.940 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.783295_189.1", 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:08.941 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:08.942 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:09.004 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:09.004 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:09.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:09.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:09.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:08Z" 2023-09-29T22:42:09.005 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:09.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.783295_189.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.783295Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:08Z" } 2023-09-29T22:42:09.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:09.006 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:09.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:09.006 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-29T22:42:14.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:14.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:14.007 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:14.054 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:14.054 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:14.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:14.085+0000 7fbaf8e30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:14.085 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:14.086+0000 7fbaf8e30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:14.100 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.101 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:14.102 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:14.103 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.822519_195.1", 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:14.104 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:14.105 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:14.106 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:14Z" 2023-09-29T22:42:14.107 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:14.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.822519_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.822519Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:14Z" } 2023-09-29T22:42:14.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:14.109 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:14.109 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:14.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.822519_195.1 2023-09-29T22:42:19.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:19.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:19.110 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:19.186 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:19.187 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:19.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:19.269+0000 7fa7b8c19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:19.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:19.270+0000 7fa7b8c19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:19.310 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:19.310 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:19.311 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:19.313 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:42:19.314 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:19.315 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:19.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:19.317 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:19Z" 2023-09-29T22:42:19.317 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:19.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:19Z" } 2023-09-29T22:42:19.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:19.318 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:19.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:19.318 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-29T22:42:24.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:24.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:24.320 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:24.365 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:24.366 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:24.400 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:24.399+0000 7f3ca7c13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:24.401 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:24.400+0000 7f3ca7c13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:24.414 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:24.414 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:24.414 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:24.415 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:24.416 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:24.417 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.825442_197.1", 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:24.418 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:24.419 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:24.420 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:24.421 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:24Z" 2023-09-29T22:42:24.422 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:24.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.825442_197.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.825442Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:24Z" } 2023-09-29T22:42:24.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:24.423 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:24.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:24.423 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.825442_197.1 2023-09-29T22:42:29.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:29.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:29.425 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:29.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:29.621 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:29.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:29.524+0000 7f76bcd36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:29.622 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:29.524+0000 7f76bcd36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:29.623 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:29.623 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:29.623 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:29.624 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:29.625 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.626 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.960282_204.1", 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:29.627 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:29.628 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:29Z" 2023-09-29T22:42:29.629 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:29.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.960282_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.960282Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:29Z" } 2023-09-29T22:42:29.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:29.630 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:29.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:29.630 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.960282_204.1 2023-09-29T22:42:34.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:34.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:34.631 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:34.685 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:34.720+0000 7efc19626880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:34.722 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:34.721+0000 7efc19626880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:34.736 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.737 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:34.738 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:34.739 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:34.740 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.842044_199.1", 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:34.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:34Z" 2023-09-29T22:42:34.743 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:34.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.842044_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.842044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:34Z" } 2023-09-29T22:42:34.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:34.745 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:34.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:34.745 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-29T22:42:39.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:39.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:39.746 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:39.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:39.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:39.836+0000 7f63e9c26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:39.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:39.837+0000 7f63e9c26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:39.851 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:39.851 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:39.852 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:39.854 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:39.855 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:39.856 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:39.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:39.858 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:39Z" 2023-09-29T22:42:39.858 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:39.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:39Z" } 2023-09-29T22:42:39.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:39.859 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:39.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:39.859 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.851988_201.1 2023-09-29T22:42:44.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:44.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:44.861 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:44.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:44.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:44.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:44.944+0000 7f1904c24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:44.945 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:44.945+0000 7f1904c24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:44.958 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:44.959 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:44.961 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:44.961 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:44.961 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:44.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:44.963 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.905539_203.1", 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:44.964 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:44.965 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:44.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:44Z" 2023-09-29T22:42:44.967 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:44.968 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.905539_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.905539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:44Z" } 2023-09-29T22:42:44.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:44.968 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:44.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:44.969 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-29T22:42:49.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:49.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:49.970 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:50.016 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:50.017 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:50.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:50.049+0000 7ff08cb36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:50.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:50.050+0000 7ff08cb36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:50.209 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:50.210 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:50.211 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:50.212 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.797765_192.1", 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.213 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:50.214 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:50.215 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:50.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:50.216 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:50Z" 2023-09-29T22:42:50.216 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:50.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.797765_192.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.797765Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:50Z" } 2023-09-29T22:42:50.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:50.217 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:50.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:50.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-29T22:42:55.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:55.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:42:55.218 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:42:55.271 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:55.272 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:55.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:55.326+0000 7f1806326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:55.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:42:55.327+0000 7f1806326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:42:55.351 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:42:55.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:55.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:55.354 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.903428_202.1", 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:55.355 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:42:55.356 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:42:55.357 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:42:55.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:42:55.358 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:42:55Z" 2023-09-29T22:42:55.358 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:42:55.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.903428_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.903428Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:42:55Z" } 2023-09-29T22:42:55.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:42:55.359 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:55.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:42:55.359 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.903428_202.1 2023-09-29T22:43:00.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:00.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:00.361 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:00.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:00.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:00.489 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:00.489+0000 7f3a65699880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:00.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:00.490+0000 7f3a65699880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:00.509 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:00.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:00.511 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:00.512 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.905539_203.1", 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:00.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:00.514 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.515 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:00.516 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:00.516 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:00.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:00.516 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:00Z" 2023-09-29T22:43:00.516 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:00.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.905539_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.905539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:00Z" } 2023-09-29T22:43:00.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:00.517 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:00.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:00.518 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-29T22:43:05.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:05.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:05.519 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:05.566 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:05.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:05.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:05.629+0000 7f5dbf136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:05.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:05.629+0000 7f5dbf136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:05.654 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:05.654 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:05.654 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:05.655 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:05.656 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.851988_201.1", 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-29T22:43:05.658 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:05.659 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:05.660 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:05.661 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:05Z" 2023-09-29T22:43:05.661 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:05.662 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.851988_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.851988Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:05Z" } 2023-09-29T22:43:05.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:05.662 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:05.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:05.662 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.851988_201.1 2023-09-29T22:43:10.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:10.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:10.790 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:10.838 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:10.838 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:10.882 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:10.882+0000 7f90b6733880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:10.883 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:10.883+0000 7f90b6733880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:10.901 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:10.901 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:10.901 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:10.902 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:10.903 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:10.904 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:10.905 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:10.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:10.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:10Z" 2023-09-29T22:43:10.908 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:10.910 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:10Z" } 2023-09-29T22:43:10.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:10.910 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:10.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:10.910 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.978724_210.1 2023-09-29T22:43:15.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:15.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:15.912 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:15.964 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:15.964 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:16.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:16.009+0000 7f84e5e44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:16.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:16.010+0000 7f84e5e44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:16.028 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:16.029 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:16.030 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:16.031 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.905539_203.1", 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:16.032 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:16.033 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:16Z" 2023-09-29T22:43:16.034 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:16.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.905539_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.905539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:16Z" } 2023-09-29T22:43:16.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:16.036 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:16.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:16.036 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-29T22:43:21.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:21.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:21.038 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:21.085 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:21.085 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:21.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:21.129+0000 7f1c257a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:21.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:21.130+0000 7f1c257a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:21.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:21.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:21.146 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.842044_199.1", 2023-09-29T22:43:21.147 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:21.148 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:21.149 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:21Z" 2023-09-29T22:43:21.150 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:21.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.842044_199.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.842044Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:21Z" } 2023-09-29T22:43:21.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:21.152 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:21.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:21.152 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-29T22:43:26.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:26.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:26.153 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:26.199 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:26.199 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:26.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:26.238+0000 7f4f2c213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:26.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:26.239+0000 7f4f2c213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:26.263 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:26.263 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:26.263 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:26.263 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:26.263 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:26.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:26.265 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:26.266 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.820374_194.1", 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.267 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:26.268 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:26Z" 2023-09-29T22:43:26.269 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:26.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.820374_194.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.820374Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:26Z" } 2023-09-29T22:43:26.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:26.271 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:26.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:26.271 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-29T22:43:31.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:31.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:31.365 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:31.413 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:31.413 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:31.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:31.453+0000 7fc905723880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:31.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:31.454+0000 7fc905723880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:31.471 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:31.471 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:31.471 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:31.471 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:31.472 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:31.473 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:31.474 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.839800_198.1", 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:31.475 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:31.476 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:31Z" 2023-09-29T22:43:31.477 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:31.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.839800_198.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.839800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:31Z" } 2023-09-29T22:43:31.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:31.479 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:31.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:31.479 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-29T22:43:36.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:36.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:43:36.480 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:43:36.527 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:36.528 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:36.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:36.570+0000 7f439e212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:36.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:43:36.571+0000 7f439e212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:43:36.586 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:43:36.586 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:43:36.586 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:36.587 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:43:36.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.589 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.849856_200.1", 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:36.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:43:36.591 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:43:36Z" 2023-09-29T22:43:36.592 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:43:36.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.849856_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.849856Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:43:36Z" } 2023-09-29T22:43:36.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:43:36.594 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:36.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:43:36.594 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-29T22:43:41.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2023-09-29T22:43:41.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:41.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:41.600 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:43:41.650 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:41.650 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:41.671 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2023-09-29T22:43:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:43:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:43:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:43:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2023-09-29T22:43:41.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:43:41.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:41.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:41.677 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:43:41.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:41.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:41.767 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e 2023-09-29T22:43:41.786 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:43:41.786 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:41.786 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 4, 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2023-09-29T22:43:41.787 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": true, 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:43:41.788 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:43:41.789 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2023-09-29T22:43:41.790 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:43:41.791 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:43:41.792 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": false, 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:43:41.793 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2023-09-29T22:43:41.794 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled_features": [ 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:43:41.795 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2885950644 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:43:41.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2464975307 2023-09-29T22:43:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:43:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:43:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1018467004 2023-09-29T22:43:41.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:43:41.798 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.799 INFO:teuthology.orchestra.run.smithi184.stdout: "user_ratelimit": { 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_ratelimit": { 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:43:41.800 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "anonymous_ratelimit": { 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:43:41.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:43:41.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:43:41.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2023-09-29T22:43:41.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:43:41.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:43:41.803 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2023-09-29T22:44:41.804 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-45 2023-09-29T22:44:41.805 DEBUG:boto:path=/dmuhsu-45/ 2023-09-29T22:44:41.805 DEBUG:boto:auth_path=/dmuhsu-45/ 2023-09-29T22:44:41.805 DEBUG:boto:Method: PUT 2023-09-29T22:44:41.805 DEBUG:boto:Path: /dmuhsu-45/ 2023-09-29T22:44:41.805 DEBUG:boto:Data: 2023-09-29T22:44:41.805 DEBUG:boto:Headers: {} 2023-09-29T22:44:41.806 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:44:41.806 DEBUG:boto:Port: 8000 2023-09-29T22:44:41.806 DEBUG:boto:Params: {} 2023-09-29T22:44:41.806 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:44:41.806 DEBUG:boto:Token: None 2023-09-29T22:44:41.807 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:44:41 GMT /dmuhsu-45/ 2023-09-29T22:44:41.807 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j1Jyd7whke0sLmUcCw97gnGNF0Y= 2023-09-29T22:44:41.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:44:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j1Jyd7whke0sLmUcCw97gnGNF0Y=', 'Content-Length': '0'} 2023-09-29T22:44:41.830 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e67ea77f7079cb6-0065175359-1abd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:44:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:44:41.831 DEBUG:boto:path=/dmuhsu-45/testredirect 2023-09-29T22:44:41.831 DEBUG:boto:auth_path=/dmuhsu-45/testredirect 2023-09-29T22:44:41.831 DEBUG:boto:Method: PUT 2023-09-29T22:44:41.832 DEBUG:boto:Path: /dmuhsu-45/testredirect 2023-09-29T22:44:41.832 DEBUG:boto:Data: 2023-09-29T22:44:41.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:44:41.832 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:44:41.832 DEBUG:boto:Port: 8000 2023-09-29T22:44:41.833 DEBUG:boto:Params: {} 2023-09-29T22:44:41.833 DEBUG:boto:Token: None 2023-09-29T22:44:41.833 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 22:44:41 GMT /dmuhsu-45/testredirect 2023-09-29T22:44:41.833 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qdCQMFFpqiPzQ7Dr0Mi2JGzeX4g= 2023-09-29T22:44:41.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:44:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qdCQMFFpqiPzQ7Dr0Mi2JGzeX4g='} 2023-09-29T22:44:41.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000645302d0fa717290-0065175359-1abd-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:44:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:44:41.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:41.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:41.843 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:44:41.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:44:41.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:44:42.007 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:44:42.024 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:44:42.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:44:42.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:44:41.827984Z" 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:44:42.026 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:44:42.027 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:44:42.028 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:42.028 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:44:42.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:42.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:42.028 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:44:42.081 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:44:42.082 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:44:42.114 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:42.114+0000 7f1592b1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:42.115 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:42.115+0000 7f1592b1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:42.129 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:44:42.129 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:44:42.129 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:44:42.129 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:44:42.129 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:44:42.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:44:42.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:42.132 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:44:42.133 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:42.134 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:44:42.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:44:42Z" 2023-09-29T22:44:42.136 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:44:42.137 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:44:42Z" } 2023-09-29T22:44:42.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:44:42.137 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:42.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:42.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:44:47.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:47.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:47.139 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:44:47.192 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:44:47.192 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:44:47.223 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:47.223+0000 7f5defe15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:47.224 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:47.224+0000 7f5defe15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:47.236 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.237 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:44:47.238 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:47.239 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:44:47.240 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:47.241 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:44:47.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:44:47Z" 2023-09-29T22:44:47.243 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:44:47.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:44:47Z" } 2023-09-29T22:44:47.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:44:47.244 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:47.244 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:47.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:44:52.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:52.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:52.246 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:44:52.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:44:52.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:44:52.331 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:52.331+0000 7f8b6aa10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:52.332 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:52.331+0000 7f8b6aa10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:52.344 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:44:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:44:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:44:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:44:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:44:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:44:52.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:44:52.351 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:44:52.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:52.351 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:44:52Z" 2023-09-29T22:44:52.351 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:44:52.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:44:52Z" } 2023-09-29T22:44:52.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:44:52.352 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:52.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:52.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:44:57.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:57.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:44:57.354 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:44:57.407 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:44:57.407 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:44:57.439 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:57.439+0000 7fcd9c929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:57.439 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:44:57.439+0000 7fcd9c929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:44:57.452 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:44:57.452 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:44:57.452 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:44:57.453 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:44:57.453 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:44:57.453 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:44:57.453 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:57.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.454 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:44:57.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:57.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:44:57.454 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:57.454 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:44:57.455 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:57.456 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:57.457 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:44:57.457 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:57.457 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:57.457 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:57.457 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:44:57.458 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:57.459 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:44:57.460 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:57.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:57.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:44:57.460 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:44:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:44:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:44:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:44:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:44:57.461 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:44:57.462 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:44:57.463 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:44:57Z" 2023-09-29T22:44:57.464 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:44:57.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:44:57Z" } 2023-09-29T22:44:57.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:44:57.465 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:57.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:44:57.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:02.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:02.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:02.467 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:02.520 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:02.520 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:02.552 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:02.551+0000 7f1fa4127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:02.552 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:02.552+0000 7f1fa4127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:02.565 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:02.566 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:02.567 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:02.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:02.570 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:02.571 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:02.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:02.572 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:02Z" 2023-09-29T22:45:02.572 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:02.573 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:02Z" } 2023-09-29T22:45:02.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:02.573 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:02.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:02.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:07.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:07.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:07.575 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:07.628 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:07.628 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:07.660 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:07.659+0000 7fd0e8536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:07.660 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:07.660+0000 7fd0e8536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:07.673 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:07.674 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:07.675 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:07Z" 2023-09-29T22:45:07.679 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:07.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:07Z" } 2023-09-29T22:45:07.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:07.681 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:07.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:07.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:12.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:12.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:12.682 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:12.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:12.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:12.767 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:12.767+0000 7fa11ca26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:12.768 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:12.768+0000 7fa11ca26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:12.780 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:12.780 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:12.781 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:12.783 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:12.784 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:12.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:12.786 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:12.787 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:12.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:12.787 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:12Z" 2023-09-29T22:45:12.787 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:12.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:12Z" } 2023-09-29T22:45:12.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:12.788 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:12.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:12.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:17.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:17.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:17.791 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:17.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:17.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:17.877 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:17.876+0000 7fb15fc13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:17.877 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:17.877+0000 7fb15fc13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:17.890 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:17.891 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:17.892 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:17.893 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:17.894 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:17.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:17Z" 2023-09-29T22:45:17.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:17.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:17Z" } 2023-09-29T22:45:17.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:17.897 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:17.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:17.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:22.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:22.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:22.899 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:22.954 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:22.954 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:22.987 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:22.987+0000 7fa9a2026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:22.987 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:22.987+0000 7fa9a2026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:23.002 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:23.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:23.004 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:23.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:23.006 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:23.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:23.008 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:22Z" 2023-09-29T22:45:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:23.010 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:22Z" } 2023-09-29T22:45:23.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:23.010 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:23.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:23.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:28.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:28.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:28.012 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:28.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:28.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:28.096 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:28.096+0000 7f08275a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:28.097 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:28.097+0000 7f08275a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:28.109 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:28Z" 2023-09-29T22:45:28.118 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:28.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:28Z" } 2023-09-29T22:45:28.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:28.119 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:28.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:28.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:33.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:33.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:33.121 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:33.176 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:33.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:33.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:33.212+0000 7f6ddc71e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:33.214 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:33.213+0000 7f6ddc71e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:33.226 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:33.227 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:33.227 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:33.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:33.227 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:33.228 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:33.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:33.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.228 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:33.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:33.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:33.230 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:33.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:33.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:33.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:33.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:33.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:33.236 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:33Z" 2023-09-29T22:45:33.236 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:33.237 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:33Z" } 2023-09-29T22:45:33.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:33.237 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:33.237 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:33.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:38.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:38.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:38.239 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:38.291 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:38.291 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:38.323 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:38.323+0000 7fd343b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:38.323 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:38.324+0000 7fd343b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:38.337 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:38.337 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:38.337 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:38.337 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:38.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:38.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027105.772402_287.1", 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:38.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:38.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:38.343 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:38Z" 2023-09-29T22:45:38.344 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:38.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027105.772402_287.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:25.772402Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:38Z" } 2023-09-29T22:45:38.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:38.345 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:38.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:38.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-29T22:45:43.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:43.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:43.346 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:43.399 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:43.400 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:43.433 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:43.433+0000 7f9505f34880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:43.433 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:45:43.434+0000 7f9505f34880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:43.446 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:43.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:43.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:43.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:45:43.451 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:45:43.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:45:43.453 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:45:43Z" 2023-09-29T22:45:43.453 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:45:43.453 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:43Z" } 2023-09-29T22:45:43.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:43.454 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:43.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:43.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:45:43.454 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:45:43.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:43.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:43.455 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:43.500 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:43.500 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:43.533 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:43.533+0000 7f07f9413880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:43.534 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:43.534+0000 7f07f9413880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:45:43.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:43.552 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:45:43.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:43.554 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:43.555 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:45:43.556 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:45:43.557 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:45:43.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:43.557 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:45:43Z" 2023-09-29T22:45:43.557 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:45:43.558 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:43Z" } 2023-09-29T22:45:43.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:43.558 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:43.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:43.558 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:45:48.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:48.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:48.562 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:48.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:48.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:48.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:48.656+0000 7f1c35536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:48.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:48.657+0000 7f1c35536880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:48.682 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:48.683 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:48.684 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.980766_211.1", 2023-09-29T22:45:48.685 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:45:48.686 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:48.687 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:45:48Z" 2023-09-29T22:45:48.688 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:45:48.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.980766_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.980766Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:48Z" } 2023-09-29T22:45:48.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:48.689 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:48.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:48.690 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-29T22:45:53.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:53.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:53.691 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:53.737 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:53.737 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:53.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:53.803+0000 7f9a50444880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:53.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:53.804+0000 7f9a50444880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:53.831 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:53.832 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:53.833 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:45:53.834 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:53.836 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:45:53Z" 2023-09-29T22:45:53.837 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:45:53.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:53Z" } 2023-09-29T22:45:53.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:53.839 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:53.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:53.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:45:58.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:58.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:45:58.840 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:45:58.888 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:45:58.888 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:45:58.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:58.941+0000 7fb37f136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:58.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:45:58.942+0000 7fb37f136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:45:58.962 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:58.963 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.964 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:58.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.975764_208.1", 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:58.966 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:45:58.967 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:45:58.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:45:58.969 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:45:58Z" 2023-09-29T22:45:58.969 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:45:58.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.975764_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.975764Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:45:58Z" } 2023-09-29T22:45:58.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:45:58.970 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:58.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:45:58.970 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-29T22:46:03.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:03.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:03.972 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:04.043 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:04.044 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:04.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:04.105+0000 7fe8cb212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:04.105 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:04.106+0000 7fe8cb212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:04.120 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:04.120 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:04.120 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:04.121 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:04.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:04.123 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:04.124 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:04.125 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:04.126 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:04.127 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:04.127 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:04.127 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:04Z" 2023-09-29T22:46:04.127 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:04.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:04Z" } 2023-09-29T22:46:04.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:04.128 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:04.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:04.128 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:46:09.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:09.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:09.130 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:09.176 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:09.176 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:09.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:09.229+0000 7f30e7e15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:09.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:09.230+0000 7f30e7e15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:09.255 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:09.255 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:09.255 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:09.255 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:09.255 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:09.256 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:09.257 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:09.258 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.259 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:09.260 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:09Z" 2023-09-29T22:46:09.261 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:09.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:09Z" } 2023-09-29T22:46:09.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:09.263 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:09.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:09.263 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:46:14.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:14.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:14.265 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:14.317 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:14.317 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:14.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:14.370+0000 7f900f0c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:14.370 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:14.371+0000 7f900f0c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:14.387 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:14.387 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:14.388 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:14.389 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:14.390 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:14.391 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.111037_224.1", 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.392 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:14.393 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:14.394 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:14.395 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:14Z" 2023-09-29T22:46:14.395 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:14.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.111037_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.111037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:14Z" } 2023-09-29T22:46:14.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.111037_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.111037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:14.396 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:14.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.111037_224.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:14.396 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.111037_224.1 2023-09-29T22:46:19.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:19.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:19.398 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:19.476 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:19.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:19.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:19.550+0000 7fe41b615880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:19.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:19.551+0000 7fe41b615880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:19.578 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:19.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:19.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:19.582 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:19.583 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:19.584 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:19.585 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:19.585 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:19.585 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:19Z" 2023-09-29T22:46:19.585 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:19.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:19Z" } 2023-09-29T22:46:19.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:19.586 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:19.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:19.586 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:46:24.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:24.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:24.588 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:24.634 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:24.635 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:24.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:24.670+0000 7f0470d36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:24.670 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:24.671+0000 7f0470d36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:24.689 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:24.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:24.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:24.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.221815_233.1", 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:24.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.221815Z", 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:24.694 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.695 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:24.696 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:24.696 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:24.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:24.696 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:24Z" 2023-09-29T22:46:24.696 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:24.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.221815_233.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.221815Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:24Z" } 2023-09-29T22:46:24.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.221815_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.221815Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:24.697 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:24.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.221815_233.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:24.698 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.221815_233.1 2023-09-29T22:46:29.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:29.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:29.700 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:29.753 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:29.753 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:29.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:29.787+0000 7fa0b9311880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:29.789 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:29.788+0000 7fa0b9311880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:29.803 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:29.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:29.805 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:29.805 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.806 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:29.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:29.808 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:29.809 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:29Z" 2023-09-29T22:46:29.810 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:29.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:29Z" } 2023-09-29T22:46:29.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:29.812 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:29.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:29.812 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.978724_210.1 2023-09-29T22:46:34.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:34.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:34.813 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:34.882 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:34.882 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:34.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:34.928+0000 7fc29fe10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:34.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:34.929+0000 7fc29fe10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:34.949 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:34.950 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:34.951 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:34.952 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:34.953 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:34.954 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:34Z" 2023-09-29T22:46:34.955 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:34.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:34Z" } 2023-09-29T22:46:34.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:34.957 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:34.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:34.957 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-29T22:46:39.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:39.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:39.958 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:40.006 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:40.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:40.048+0000 7f8a07e24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:40.050 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:40.050+0000 7f8a07e24880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:40.073 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:40.074 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:40.075 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:40.076 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.110230_223.1", 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:40.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:40.078 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:40.079 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:40Z" 2023-09-29T22:46:40.080 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:40.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.110230_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.110230Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:40Z" } 2023-09-29T22:46:40.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:40.081 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:40.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:40.081 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-29T22:46:45.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:45.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:45.082 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:45.131 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:45.131 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:45.166 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:45.166+0000 7f423b80a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:45.167 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:45.167+0000 7f423b80a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:45.184 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:45.184 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:45.184 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:45.184 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:45.186 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:45.187 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:45.188 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:46:45.189 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:45Z" 2023-09-29T22:46:45.192 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:45.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:45Z" } 2023-09-29T22:46:45.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:45.193 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:45.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:45.194 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-29T22:46:50.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:50.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:50.195 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:50.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:50.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:50.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:50.324+0000 7fdeb2213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:50.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:50.326+0000 7fdeb2213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:50.344 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:50.344 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:50.345 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:50.345 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:50.345 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:50.345 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:50.345 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:50.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.346 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:50.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:50.346 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:50.347 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:50.347 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:50.347 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:50.347 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:50.347 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:50.348 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:50.348 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:50.348 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:50.348 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:50.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:50.349 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:50.349 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:50.349 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:50.349 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:50.350 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:50.350 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:50.350 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:50.350 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:50.351 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:50.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:50.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:50.351 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:50.352 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:50.352 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:50.352 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.975764_208.1", 2023-09-29T22:46:50.352 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:50.352 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:50.353 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:50.353 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-29T22:46:50.353 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:50.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:50.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.354 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:50.354 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:50.354 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:50.354 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:50.355 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:50.355 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:50.355 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:50.355 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:50.355 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:50.356 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:50.356 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:50.356 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:50.356 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:50.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.357 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:50.357 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:50.357 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:50.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:50.357 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:50Z" 2023-09-29T22:46:50.358 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:50.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.975764_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.975764Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:50Z" } 2023-09-29T22:46:50.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:50.359 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:50.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:50.360 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-29T22:46:55.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:55.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:46:55.362 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:46:55.409 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:55.409 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:55.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:55.448+0000 7f48c7136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:55.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:46:55.449+0000 7f48c7136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:46:55.462 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:46:55.462 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:46:55.462 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:46:55.462 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:46:55.462 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:55.463 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:46:55.464 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.980766_211.1", 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:55.466 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:46:55.467 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:46:55.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:46:55.469 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:46:55Z" 2023-09-29T22:46:55.469 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:46:55.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.980766_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.980766Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:46:55Z" } 2023-09-29T22:46:55.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:46:55.470 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:55.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:46:55.470 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-29T22:47:00.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:00.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:00.471 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:00.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:00.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:00.575 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:00.575+0000 7fe9648b8880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:00.576 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:00.576+0000 7fe9648b8880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:00.589 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:00.590 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:00.590 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:00.590 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:00.590 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:00.591 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:00.592 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:00.592 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:00.592 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:00.592 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:00.592 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.593 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:00.594 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:00.594 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:00.594 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:00.594 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:00.594 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:00.595 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.003827_212.1", 2023-09-29T22:47:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:00.597 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.598 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:00.598 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:00.598 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:00.598 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:00.598 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:00.599 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:00.600 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:00.600 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:00.600 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.600 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:00.600 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:00.601 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:00.601 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:00.601 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:00Z" 2023-09-29T22:47:00.601 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:00.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.003827_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.003827Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:00Z" } 2023-09-29T22:47:00.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:00.603 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:00.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:00.603 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.003827_212.1 2023-09-29T22:47:05.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:05.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:05.605 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:05.665 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:05.666 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:05.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:05.726+0000 7f1a2982e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:05.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:05.727+0000 7f1a2982e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:05.747 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:05.748 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:05.749 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:05.750 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026484.251433_236.1", 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:05.751 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:05.752 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:05.753 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:05Z" 2023-09-29T22:47:05.754 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:05.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026484.251433_236.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:04.251433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:05Z" } 2023-09-29T22:47:05.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.251433_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.251433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:05.756 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:05.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.251433_236.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:05.756 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.251433_236.1 2023-09-29T22:47:10.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:10.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:10.757 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:10.803 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:10.803 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:10.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:10.836+0000 7fc295cb6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:10.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:10.837+0000 7fc295cb6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:10.857 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.858 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:10.859 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:10.860 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:47:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:10.862 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:10Z" 2023-09-29T22:47:10.864 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:10.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:10Z" } 2023-09-29T22:47:10.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:10.865 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:10.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:10.866 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:47:15.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:15.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:15.867 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:15.916 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:15.916 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:15.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:15.961+0000 7f9ca6136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:15.961 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:15.962+0000 7f9ca6136880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:15.980 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:15.980 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:15.981 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:15.982 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:15.983 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.009067_215.1", 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:15.984 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:15.985 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:15.986 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:15Z" 2023-09-29T22:47:15.987 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:15.987 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.009067_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.009067Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:15Z" } 2023-09-29T22:47:15.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:15.988 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:15.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:15.988 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-29T22:47:20.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:20.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:20.990 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:21.039 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:21.039 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:21.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:21.092+0000 7f8596927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:21.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:21.092+0000 7f8596927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:21.114 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:21.115 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:21.116 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:21.117 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.009067_215.1", 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.118 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:21.119 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:21Z" 2023-09-29T22:47:21.120 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:21.121 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.009067_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.009067Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:21Z" } 2023-09-29T22:47:21.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:21.122 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:21.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:21.122 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-29T22:47:26.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:26.123 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:26.174 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:26.174 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:26.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:26.205+0000 7fdd1fa67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:26.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:26.206+0000 7fdd1fa67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:26.227 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:26.227 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:26.227 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:26.227 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:26.228 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:26.230 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.023985_216.1", 2023-09-29T22:47:26.231 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:26.232 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:26.233 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:26Z" 2023-09-29T22:47:26.234 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:26.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.023985_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.023985Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:26Z" } 2023-09-29T22:47:26.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:26.236 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:26.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:26.236 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-29T22:47:31.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:31.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:31.237 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:31.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:31.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:31.338 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:31.334+0000 7fcc4ed1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:31.343 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:31.341+0000 7fcc4ed1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:31.403 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:31.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:31.405 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:31.406 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:31.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.977901_209.1", 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:31.408 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:31.409 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:31.410 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:31Z" 2023-09-29T22:47:31.411 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:31.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.977901_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.977901Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:31Z" } 2023-09-29T22:47:31.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:31.413 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:31.413 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:31.413 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-29T22:47:36.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:36.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:36.414 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:36.461 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:36.461 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:36.493 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:36.493+0000 7fbb53f23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:36.494 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:36.493+0000 7fbb53f23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.009067_215.1", 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:36Z" 2023-09-29T22:47:36.514 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:36.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.009067_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.009067Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:36Z" } 2023-09-29T22:47:36.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:36.516 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:36.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:36.516 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-29T22:47:41.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:41.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:41.517 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:41.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:41.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:41.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:41.618+0000 7f6ecd626880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:41.620 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:41.619+0000 7f6ecd626880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:41.637 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:41.638 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:41.639 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:41.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:41.641 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:41.642 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:41.643 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:41Z" 2023-09-29T22:47:41.644 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:41.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:41Z" } 2023-09-29T22:47:41.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:41.646 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:41.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:41.646 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:47:46.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:46.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:46.647 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:46.700 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:46.700 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:46.749 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:46.749+0000 7f84ead29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:46.750 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:46.750+0000 7f84ead29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:46.777 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:46.777 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:46.777 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:46.777 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:46.778 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:46.779 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:46.781 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:46.782 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:46Z" 2023-09-29T22:47:46.783 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:46.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:46Z" } 2023-09-29T22:47:46.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:46.785 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:46.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:46.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-29T22:47:51.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:51.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:51.786 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:51.833 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:51.870+0000 7f33e3c9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:51.870 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:51.871+0000 7f33e3c9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:51.886 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:51.887 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:51.888 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.977901_209.1", 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:51.890 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:51.891 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.892 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:51.893 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:51Z" 2023-09-29T22:47:51.893 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:51.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.977901_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.977901Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:51Z" } 2023-09-29T22:47:51.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:51.894 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:51.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:51.895 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-29T22:47:56.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:56.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:47:56.896 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:47:56.949 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:56.949 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:56.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:56.993+0000 7f9b4931e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:56.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:47:56.994+0000 7f9b4931e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:57.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:47:57.011 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:47:57.012 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:57.013 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:47:57.014 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:47:57.015 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:47:56Z" 2023-09-29T22:47:57.016 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:47:57.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:47:56Z" } 2023-09-29T22:47:57.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:47:57.017 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:57.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:47:57.017 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:48:02.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:02.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:02.018 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:02.094 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:02.182 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:02.182+0000 7fb165815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:02.183 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:02.183+0000 7fb165815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:02.198 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:02.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:02.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:02.201 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.009067_215.1", 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:02.203 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:02.204 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:02.205 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:02.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:02.205 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:02Z" 2023-09-29T22:48:02.205 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:02.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.009067_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.009067Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:02Z" } 2023-09-29T22:48:02.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:02.206 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:02.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:02.206 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-29T22:48:07.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:07.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:07.208 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:07.259 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:07.259 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:07.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:07.305+0000 7f1228ac0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:07.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:07.306+0000 7f1228ac0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:07.330 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:07.330 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:07.331 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:07.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:07.333 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.977901_209.1", 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:07.334 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:07.335 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.336 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:07.337 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:07.337 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:07.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:07.337 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:07Z" 2023-09-29T22:48:07.337 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:07.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.977901_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.977901Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:07Z" } 2023-09-29T22:48:07.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:07.338 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:07.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:07.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-29T22:48:12.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:12.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:12.340 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:12.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:12.389 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:12.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:12.423+0000 7f0b267a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:12.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:12.424+0000 7f0b267a2880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:12.438 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:12.438 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:12.438 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:12.438 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:12.440 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:12.441 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.978724_210.1", 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.442 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:12.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:12Z" 2023-09-29T22:48:12.444 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:12.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.978724_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.978724Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:12Z" } 2023-09-29T22:48:12.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:12.445 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:12.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:12.446 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.978724_210.1 2023-09-29T22:48:17.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:17.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:17.449 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:17.528 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:17.529 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:17.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:17.618+0000 7f4625812880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:17.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:17.619+0000 7f4625812880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:17.634 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:17.634 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:17.635 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:17.636 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.637 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.023985_216.1", 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:17.638 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:17.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:17Z" 2023-09-29T22:48:17.640 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:17.641 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.023985_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.023985Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:17Z" } 2023-09-29T22:48:17.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:17.642 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:17.642 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:17.642 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-29T22:48:22.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:22.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:22.643 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:22.691 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:22.692 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:22.725+0000 7fed9c9af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:22.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:22.725+0000 7fed9c9af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:22.739 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:22.739 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:22.739 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:22.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:22.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:22.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026484.251433_236.1", 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:22.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:22.744 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:22.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:22Z" 2023-09-29T22:48:22.746 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:22.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026484.251433_236.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:04.251433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:22Z" } 2023-09-29T22:48:22.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.251433_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.251433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:22.747 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:22.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.251433_236.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:22.748 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.251433_236.1 2023-09-29T22:48:27.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:27.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:27.749 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:27.801 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:27.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:27.835+0000 7fcd4ad4b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:27.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:27.836+0000 7fcd4ad4b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:27.852 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:27.853 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.965243_207.1", 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:27.856 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:27.857 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:27.858 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:27.859 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:27Z" 2023-09-29T22:48:27.859 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:27.860 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.965243_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.965243Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:27Z" } 2023-09-29T22:48:27.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:27.860 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:27.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:27.861 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-29T22:48:32.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:32.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:32.862 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:32.936 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:32.936 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:32.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:32.978+0000 7f1847929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:32.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:32.979+0000 7f1847929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:32.997 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:32.997 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:32.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:32.999 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:33.000 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:33.001 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.977901_209.1", 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:33.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:33.003 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:33.004 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:33.005 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:33.005 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:32Z" 2023-09-29T22:48:33.005 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:33.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.977901_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.977901Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:32Z" } 2023-09-29T22:48:33.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:33.006 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:33.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:33.007 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-29T22:48:38.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:38.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:38.008 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:38.061 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:38.061 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:38.099 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:38.098+0000 7f7d82e9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:38.100 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:38.099+0000 7f7d82e9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:38.115 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:38.115 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:38.115 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:38.116 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:38.117 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:38.118 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006025_213.1", 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:38.119 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:38.120 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:38.121 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:38.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:38.122 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:38Z" 2023-09-29T22:48:38.122 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:38.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006025_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006025Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:38Z" } 2023-09-29T22:48:38.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006025_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:38.123 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:38.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006025_213.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:38.123 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006025_213.1 2023-09-29T22:48:43.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:43.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:43.125 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:43.174 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:43.175 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:43.221+0000 7efc17929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:43.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:43.222+0000 7efc17929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:43.237 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:43.237 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:43.238 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:43.239 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:43.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:48:43.241 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:43.242 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:43.243 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:43.244 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:43Z" 2023-09-29T22:48:43.244 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:43.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:43Z" } 2023-09-29T22:48:43.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:43.245 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:43.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:43.245 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:48:48.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:48.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:48.247 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:48.315 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:48.315 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:48.361 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:48.361+0000 7f7d95737880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:48.362 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:48.362+0000 7f7d95737880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:48.377 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:48.378 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:48.379 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.026351_217.1", 2023-09-29T22:48:48.380 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:48.381 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.382 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:48.383 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:48.383 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:48.383 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:48.383 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:48Z" 2023-09-29T22:48:48.383 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:48.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.026351_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.026351Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:48Z" } 2023-09-29T22:48:48.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:48.384 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:48.384 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:48.384 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-29T22:48:53.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:53.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:53.386 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:53.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:53.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:53.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:53.476+0000 7f0d62726880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:53.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:53.477+0000 7f0d62726880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:53.490 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:53.490 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:53.490 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:53.491 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:53.492 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.980766_211.1", 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-29T22:48:53.494 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:53.496 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:53.497 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:53Z" 2023-09-29T22:48:53.497 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:53.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.980766_211.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.980766Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:53Z" } 2023-09-29T22:48:53.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:53.498 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:53.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:53.498 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-29T22:48:58.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:58.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:48:58.500 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:48:58.553 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:58.553 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:58.589 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:58.589+0000 7f0090133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:58.590 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:48:58.590+0000 7f0090133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:48:58.605 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:48:58.605 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:48:58.605 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:48:58.605 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:48:58.605 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:48:58.606 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:58.607 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:48:58.608 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.609 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:48:58.610 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:48:58Z" 2023-09-29T22:48:58.611 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:48:58.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:48:58Z" } 2023-09-29T22:48:58.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:48:58.613 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:58.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:48:58.613 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:49:03.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:03.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:03.614 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:03.660 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:03.661 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:03.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:03.693+0000 7fdb69026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:03.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:03.694+0000 7fdb69026880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:03.707 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.708 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:03.709 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:03.710 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:03.711 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.127019_228.1", 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.127019Z", 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:03.712 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:03.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:03Z" 2023-09-29T22:49:03.714 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:03.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.127019_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.127019Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:03Z" } 2023-09-29T22:49:03.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.127019_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.127019Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:03.715 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:03.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.127019_228.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:03.716 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.127019_228.1 2023-09-29T22:49:08.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:08.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:08.717 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:08.768 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:08.768 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:08.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:08.803+0000 7f94f7537880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:08.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:08.804+0000 7f94f7537880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:08.818 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:08.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:08.820 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:08.821 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.023985_216.1", 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:08.822 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:08.823 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.824 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:08.825 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:08.825 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:08.825 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:08.825 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:08Z" 2023-09-29T22:49:08.825 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:08.826 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.023985_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.023985Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:08Z" } 2023-09-29T22:49:08.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:08.827 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:08.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:08.827 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-29T22:49:13.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:13.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:13.828 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:13.876 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:13.876 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:13.908 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:13.908+0000 7f60a9a14880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:13.909 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:13.909+0000 7f60a9a14880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:13.926 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.023985_216.1", 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:13.927 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:13.928 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:13Z" 2023-09-29T22:49:13.929 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:13.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.023985_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.023985Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:13Z" } 2023-09-29T22:49:13.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:13.931 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:13.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:13.931 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-29T22:49:18.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:18.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:18.932 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:18.981 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:18.981 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:19.020 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:19.020+0000 7f5afd619880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:19.021 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:19.021+0000 7f5afd619880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:19.035 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:19.036 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:19.037 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:19.038 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026482.975764_208.1", 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:19.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:19.040 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:19Z" 2023-09-29T22:49:19.041 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:19.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026482.975764_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:02.975764Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:19Z" } 2023-09-29T22:49:19.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:19.042 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:19.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:19.043 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-29T22:49:24.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:24.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:24.044 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:24.094 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:24.094 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:24.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:24.129+0000 7f2765e26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:24.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:24.130+0000 7f2765e26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:24.143 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:24.143 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:24.143 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:24.143 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:24.143 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:24.144 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:24.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.006853_214.1", 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:24.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:24.148 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:24Z" 2023-09-29T22:49:24.149 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:24.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.006853_214.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.006853Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:24Z" } 2023-09-29T22:49:24.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:24.151 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:24.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:24.151 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-29T22:49:29.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:29.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:29.152 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:29.198 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:29.199 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:29.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:29.239+0000 7f66b654b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:29.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:29.240+0000 7f66b654b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:29.260 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:29.260 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:29.260 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:29.260 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:29.261 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:29.262 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:29.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:49:29.264 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:29.265 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:29.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:29.267 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:29Z" 2023-09-29T22:49:29.267 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:29.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:29Z" } 2023-09-29T22:49:29.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:29.268 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:29.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:29.268 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:49:34.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:34.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:34.269 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:34.316 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:34.316 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:34.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:34.364+0000 7fa059811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:34.365 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:34.364+0000 7fa059811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:34.386 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:34.387 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:34.388 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:34.389 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:34.390 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:34.391 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:34Z" 2023-09-29T22:49:34.392 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:34.393 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:34Z" } 2023-09-29T22:49:34.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:34.393 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:34.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:34.394 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:49:39.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:39.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:39.395 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:39.474 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:39.474 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.536 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:39.536+0000 7f8d29c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:39.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:39.536+0000 7f8d29c10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:39Z" 2023-09-29T22:49:39.558 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:39.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:39Z" } 2023-09-29T22:49:39.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:39.559 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:39.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:39.559 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:49:44.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:44.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:44.561 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:44.607 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:44.608 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:44.641+0000 7f954769d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:44.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:44.641+0000 7f954769d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:44.655 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:44.656 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:44.657 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:44.658 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.110230_223.1", 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:44.659 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:44.660 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:44.661 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:44.662 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:44Z" 2023-09-29T22:49:44.662 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:44.662 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.110230_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.110230Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:44Z" } 2023-09-29T22:49:44.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:44.663 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:44.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:44.663 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-29T22:49:49.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:49.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:49.664 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:49.716 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:49.717 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:49.755 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:49.755+0000 7ff329c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:49.756 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:49.756+0000 7ff329c8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:49.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:49.772 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:49.773 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.110230_223.1", 2023-09-29T22:49:49.774 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:49.775 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.776 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:49.777 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:49.777 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:49.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:49.777 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:49Z" 2023-09-29T22:49:49.777 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:49.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.110230_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.110230Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:49Z" } 2023-09-29T22:49:49.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:49.778 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:49.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:49.778 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-29T22:49:54.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:54.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:54.780 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:54.837 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:54.838 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:54.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:54.892+0000 7f26b65af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:54.893 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:49:54.893+0000 7f26b65af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:49:54.913 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:49:54.914 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:54.915 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:49:54.916 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:54.917 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:49:54.918 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.919 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:49:54.920 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:49:54.920 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:49:54.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:49:54.920 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:49:54Z" 2023-09-29T22:49:54.920 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:49:54.921 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:49:54Z" } 2023-09-29T22:49:54.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:49:54.921 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:54.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:49:54.922 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:49:59.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:59.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:59.923 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:59.971 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:59.971 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:00.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:00.018+0000 7fd98849d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:00.018 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:00.019+0000 7fd98849d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:00.032 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:00.033 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:00.034 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:00.035 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.216959_231.1", 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:00.036 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.216959Z", 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:00.037 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:00.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:00.039 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:00Z" 2023-09-29T22:50:00.039 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:00.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.216959_231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.216959Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:00Z" } 2023-09-29T22:50:00.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.216959_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.216959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:00.040 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:00.040 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.216959_231.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:00.040 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.216959_231.1 2023-09-29T22:50:05.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:05.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:05.041 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:05.093 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:05.093 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:05.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:05.126+0000 7f3dba329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:05.127 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:05.127+0000 7f3dba329880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:05.142 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:05.142 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:05.142 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:05.142 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:05.142 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:05.143 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:05.144 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:05.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:50:05.146 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:05.147 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:05.148 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:05Z" 2023-09-29T22:50:05.149 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:05.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:05Z" } 2023-09-29T22:50:05.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:05.150 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:05.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:05.150 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:50:10.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:10.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:10.152 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:10.198 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:10.198 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:10.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:10.242+0000 7f6360813880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:10.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:10.243+0000 7f6360813880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:10.259 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:10.261 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:10.262 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.201655_230.1", 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.201655Z", 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.263 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:10.264 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:10Z" 2023-09-29T22:50:10.265 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:10.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.201655_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.201655Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:10Z" } 2023-09-29T22:50:10.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.201655_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.201655Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:10.267 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:10.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.201655_230.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:10.267 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.201655_230.1 2023-09-29T22:50:15.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:15.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:15.268 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:15.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:15.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:15.383 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:15.383+0000 7fea31844880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:15.384 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:15.384+0000 7fea31844880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:15.413 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:15.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:15.415 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:15.416 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:15.417 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:15.418 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:15.419 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:15.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:15.420 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:15Z" 2023-09-29T22:50:15.420 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:15.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:15Z" } 2023-09-29T22:50:15.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:15.421 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:15.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:15.421 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:50:20.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:20.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:20.423 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:20.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:20.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:20.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:20.548+0000 7fd7b5526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:20.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:20.551+0000 7fd7b5526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:20.617 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:20.618 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:20.619 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:20.620 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:20.621 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:20.622 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:20Z" 2023-09-29T22:50:20.623 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:20.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:20Z" } 2023-09-29T22:50:20.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:20.624 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:20.624 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:20.624 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:50:25.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:25.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:25.626 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:25.672 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:25.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:25.704+0000 7fe400123880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:25.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:25.705+0000 7fe400123880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:25.729 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:25.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:25.732 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026484.254025_237.1", 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:25.733 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:04.254025Z", 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:25.734 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:25.735 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:25.736 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:25.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:25.736 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:25Z" 2023-09-29T22:50:25.736 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:25.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026484.254025_237.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:04.254025Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:25Z" } 2023-09-29T22:50:25.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.254025_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.254025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:25.737 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:25.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.254025_237.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:25.737 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.254025_237.1 2023-09-29T22:50:30.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:30.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:30.739 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:30.786 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:30.786 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:30.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:30.818+0000 7fb3b2929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:30.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:30.819+0000 7fb3b2929880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:30.833 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:30.833 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:30.834 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:30.836 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.111037_224.1", 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:30.837 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:30.838 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:30.839 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:30.840 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:30.840 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:30.840 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:30Z" 2023-09-29T22:50:30.840 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:30.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.111037_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.111037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:30Z" } 2023-09-29T22:50:30.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.111037_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.111037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:30.841 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:30.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.111037_224.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:30.841 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.111037_224.1 2023-09-29T22:50:35.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:35.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:35.843 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:35.890 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:35.890 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:35.931 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:35.931+0000 7f109a013880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:35.932 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:35.932+0000 7f109a013880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:35.946 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:35.946 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:35.946 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:35.946 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:35.946 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:35.947 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:35.948 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.950 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:35.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:35Z" 2023-09-29T22:50:35.952 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:35.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:35Z" } 2023-09-29T22:50:35.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:35.954 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:35.954 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:35.954 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:50:40.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:40.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:40.955 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:41.006 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:41.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:41.048+0000 7f935bb31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:41.049 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:41.049+0000 7f935bb31880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:41.065 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.066 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:41.067 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:41.068 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.113101_225.1", 2023-09-29T22:50:41.069 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:41.070 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:41.071 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:41Z" 2023-09-29T22:50:41.072 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:41.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.113101_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.113101Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:41Z" } 2023-09-29T22:50:41.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.113101_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.113101Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:41.073 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:41.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.113101_225.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:41.074 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.113101_225.1 2023-09-29T22:50:46.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:46.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:46.075 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:46.128 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:46.129 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:46.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:46.181+0000 7f54e8412880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:46.184 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:46.184+0000 7f54e8412880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:46.240 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:46.240 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:46.240 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.241 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:46.242 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:46.243 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:46.244 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:46.245 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:46.246 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:46.247 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:46Z" 2023-09-29T22:50:46.248 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:46.249 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:46Z" } 2023-09-29T22:50:46.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:46.250 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:46.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:46.250 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:50:51.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2023-09-29T22:50:51.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.256 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:51.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:51.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:51.334 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:50:51.346 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:50:51.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.346 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:51.347 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:50:51.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.347 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:51.347 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:50:51.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.348 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:50:51.348 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:44:41.827984Z" 2023-09-29T22:50:51.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.349 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.349 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:51.349 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:50:51.349 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.349 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:50:51.350 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:51.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:50:51.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.352 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:51.405 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:51.405 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:51.436 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:50:51.436+0000 7f6fc1336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:51.437 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:50:51.437+0000 7f6fc1336880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:51.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:50:51.450 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:50:51.451 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:50:51.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:50:51.453 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:50:51.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:50:51.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:50:51.456 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:50:51.457 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:50:51.458 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:50:51.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:50:51.458 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:50:51Z" 2023-09-29T22:50:51.458 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:50:51.459 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:51Z" } 2023-09-29T22:50:51.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:51.459 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:51.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:51.460 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:50:51.460 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:50:51.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:51.460 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:51.506 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:51.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:51.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:51.547+0000 7faa66922880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:51.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:51.548+0000 7faa66922880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:51.563 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:51.563 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:51.563 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:51.564 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.565 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:51.566 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:51.567 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.265680_238.1", 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.265680Z", 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:51.568 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:51.569 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:51.570 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:51.571 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:51.571 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:51Z" 2023-09-29T22:50:51.571 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:51.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.265680_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.265680Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:51Z" } 2023-09-29T22:50:51.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.265680_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.265680Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:51.572 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:51.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.265680_238.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:51.573 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.265680_238.1 2023-09-29T22:50:56.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:56.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:56.574 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:56.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:56.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:56.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:56.674+0000 7f5726213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:56.675 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:50:56.675+0000 7f5726213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:56.736 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:50:56.736 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:50:56.736 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:50:56.736 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:56.737 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:50:56.738 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.201655_230.1", 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.201655Z", 2023-09-29T22:50:56.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:50:56.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:50:56.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:50:56.743 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:50:56Z" 2023-09-29T22:50:56.743 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:50:56.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.201655_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.201655Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:56Z" } 2023-09-29T22:50:56.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.201655_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.201655Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:50:56.744 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:56.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.201655_230.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:50:56.744 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.201655_230.1 2023-09-29T22:51:01.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:01.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:01.746 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:01.794 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:01.794 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:01.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:01.829+0000 7f798e011880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:01.830 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:01.830+0000 7f798e011880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:01.845 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.113101_225.1", 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:01Z" 2023-09-29T22:51:01.852 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:01.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.113101_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.113101Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:01Z" } 2023-09-29T22:51:01.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.113101_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.113101Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:01.853 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:01.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.113101_225.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:01.853 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.113101_225.1 2023-09-29T22:51:06.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:06.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:06.855 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:06.908 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:06.908 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:06.946 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:06.947+0000 7ff85fb29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:06.947 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:06.947+0000 7ff85fb29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:06.962 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026797.355227_264.1", 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:33:17.355227Z", 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:06Z" 2023-09-29T22:51:06.969 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:06.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026797.355227_264.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:17.355227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:06Z" } 2023-09-29T22:51:06.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026797.355227_264.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:17.355227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:06.970 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:06.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.355227_264.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:06.970 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026797.355227_264.1 2023-09-29T22:51:11.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:11.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:11.972 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:12.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:12.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:12.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:12.056+0000 7f6b88419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:12.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:12.057+0000 7f6b88419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:12.076 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:12.077 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:12.078 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:12.079 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.268047_239.1", 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.268047Z", 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:12.080 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:12.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:12Z" 2023-09-29T22:51:12.082 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:12.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.268047_239.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.268047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:12Z" } 2023-09-29T22:51:12.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.268047_239.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.268047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:12.084 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:12.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.268047_239.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:12.084 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.268047_239.1 2023-09-29T22:51:17.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:17.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:17.085 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:17.137 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:17.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:17.189 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:17.189+0000 7f366fb23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:17.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:17.190+0000 7f366fb23880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:17.205 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:17.205 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:17.205 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:17.206 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:17.207 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:17.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.108009_222.1", 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.108009Z", 2023-09-29T22:51:17.209 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:17.210 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:17.211 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:17.212 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:17.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:17.212 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:17Z" 2023-09-29T22:51:17.212 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:17.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.108009_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.108009Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:17Z" } 2023-09-29T22:51:17.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.108009_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.108009Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:17.213 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:17.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.108009_222.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:17.214 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.108009_222.1 2023-09-29T22:51:22.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:22.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:22.215 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:22.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:22.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:22.315+0000 7f63c391c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:22.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:22.316+0000 7f63c391c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:22.340 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:22.340 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:22.340 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:22.340 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:22.341 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:22.342 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:22.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:51:22.344 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:22.345 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:22.346 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:22Z" 2023-09-29T22:51:22.347 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:22.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:22Z" } 2023-09-29T22:51:22.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:22.348 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:22.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:22.348 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:51:27.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:27.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:27.350 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:27.398 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:27.399 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:27.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:27.442+0000 7f34ff40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:27.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:27.444+0000 7f34ff40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:27.462 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:27.462 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:27.462 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:27.463 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:27.464 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:27.466 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:27.467 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:27.468 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:27Z" 2023-09-29T22:51:27.469 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:27.469 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:27Z" } 2023-09-29T22:51:27.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:27.470 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:27.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:27.470 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:51:32.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:32.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:32.471 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:32.522 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:32.522 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:32.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:32.568+0000 7f54c5b27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:32.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:32.569+0000 7f54c5b27880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:32.583 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:32.584 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:32.585 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:32.586 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:32.587 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:32.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:32.589 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:32Z" 2023-09-29T22:51:32.590 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:32.591 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:32Z" } 2023-09-29T22:51:32.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:32.591 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:32.592 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:32.592 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:51:37.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:37.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:37.593 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:37.669 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:37.670 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:37.725+0000 7f85e7415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:37.726 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:37.726+0000 7f85e7415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:37.740 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:37.740 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:37.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:37.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:37.744 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:37.745 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:37.746 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:37.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:37.747 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:37Z" 2023-09-29T22:51:37.747 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:37.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:37Z" } 2023-09-29T22:51:37.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:37.748 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:37.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:37.748 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:51:42.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:42.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:42.749 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:42.798 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:42.799 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:42.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:42.833+0000 7f50c8613880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:42.834 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:42.834+0000 7f50c8613880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:42.849 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:42.849 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:42.849 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:42.849 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:42.850 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:42.851 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:42.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:51:42.853 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:42.854 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:42.855 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:42.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:42.856 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:42Z" 2023-09-29T22:51:42.856 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:42.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:42Z" } 2023-09-29T22:51:42.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:42.857 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:42.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:42.857 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:51:47.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:47.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:47.858 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:47.908 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:47.908 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:47.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:47.941+0000 7f6d99f1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:47.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:47.942+0000 7f6d99f1e880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:47.965 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.966 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:47.967 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:47.968 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:47.969 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.026351_217.1", 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:47.970 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:47.971 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:47.972 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:47.973 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:47Z" 2023-09-29T22:51:47.974 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:47.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.026351_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.026351Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:47Z" } 2023-09-29T22:51:47.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:47.975 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:47.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:47.975 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-29T22:51:52.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:52.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:52.977 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:53.026 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:53.026 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:53.063 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:53.063+0000 7f3919c9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:53.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:53.064+0000 7f3919c9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:53.081 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:53.082 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:53.083 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.084 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.113101_225.1", 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:53.086 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:53.087 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:53.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:53Z" 2023-09-29T22:51:53.089 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:53.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.113101_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.113101Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:53Z" } 2023-09-29T22:51:53.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.113101_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.113101Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:53.091 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:53.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.113101_225.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:53.091 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.113101_225.1 2023-09-29T22:51:58.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:58.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:58.092 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:58.142 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:58.142 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:58.180 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:58.180+0000 7fcb12c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:58.181 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:51:58.181+0000 7fcb12c67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:58.194 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:51:58.195 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:58.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:58.197 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:51:58.198 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:51:58.199 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:51:58.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:51:58Z" 2023-09-29T22:51:58.201 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:51:58.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:58Z" } 2023-09-29T22:51:58.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:51:58.202 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:58.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:51:58.203 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:52:03.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:03.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:03.204 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:03.256 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:03.256 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:03.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:03.303+0000 7fd2cc936880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:03.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:03.304+0000 7fd2cc936880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:03.321 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:03.322 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:03.323 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:03.324 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.265680_238.1", 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.265680Z", 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:03.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:03Z" 2023-09-29T22:52:03.327 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:03.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.265680_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.265680Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:03Z" } 2023-09-29T22:52:03.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.265680_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.265680Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:03.329 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:03.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.265680_238.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:03.329 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.265680_238.1 2023-09-29T22:52:08.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:08.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:08.331 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:08.377 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:08.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:08.427+0000 7f74f968b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:08.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:08.428+0000 7f74f968b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:08.450 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:08.450 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:08.451 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:08.452 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:08.453 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.454 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.026351_217.1", 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-29T22:52:08.455 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:08.456 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.457 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:08.458 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:08.458 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:08.458 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:08.458 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:08Z" 2023-09-29T22:52:08.458 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:08.459 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.026351_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.026351Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:08Z" } 2023-09-29T22:52:08.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:08.460 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:08.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:08.460 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-29T22:52:13.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:13.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:13.462 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:13.514 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:13.515 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:13.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:13.559+0000 7f4871413880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:13.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:13.560+0000 7f4871413880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:13.577 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:13.577 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:13.577 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:13.578 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:13.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.281741_240.1", 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.281741Z", 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:13.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:13.582 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:13.583 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:13Z" 2023-09-29T22:52:13.584 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:13.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.281741_240.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.281741Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:13Z" } 2023-09-29T22:52:13.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.281741_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.281741Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:13.585 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:13.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.281741_240.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:13.585 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.281741_240.1 2023-09-29T22:52:18.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:18.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:18.587 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:18.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:18.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:18.668 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:18.668+0000 7f8978c30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:18.669 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:18.669+0000 7f8978c30880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:18.685 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.686 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:18.687 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:18.688 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.689 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:18.690 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:18.690 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:18.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:18.690 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:18Z" 2023-09-29T22:52:18.690 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:18.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:18Z" } 2023-09-29T22:52:18.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:18.691 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:18.692 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:18.692 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:52:23.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:23.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:23.693 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:23.742 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:23.742 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:23.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:23.776+0000 7f55e228b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:23.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:23.777+0000 7f55e228b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:23.801 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:23.802 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:23.803 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:23.804 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.108009_222.1", 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.108009Z", 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.805 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:23.806 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:23Z" 2023-09-29T22:52:23.807 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:23.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.108009_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.108009Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:23Z" } 2023-09-29T22:52:23.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.108009_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.108009Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:23.809 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:23.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.108009_222.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:23.809 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.108009_222.1 2023-09-29T22:52:28.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:28.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:28.810 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:28.870 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:28.871 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:28.910 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:28.909+0000 7f5481419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:28.911 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:28.910+0000 7f5481419880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:28.929 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:28.930 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:28.931 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:28.932 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.026351_217.1", 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:28.933 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:28.934 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:28Z" 2023-09-29T22:52:28.935 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:28.936 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.026351_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.026351Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:28Z" } 2023-09-29T22:52:28.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:28.937 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:28.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:28.937 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-29T22:52:33.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:33.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:33.938 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:33.991 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:33.991 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:34.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:34.027+0000 7f75bd013880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:34.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:34.028+0000 7f75bd013880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:34.043 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.044 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:34.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:34.046 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:34.047 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:34.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:34.049 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:34.050 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:34.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:34.051 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:34Z" 2023-09-29T22:52:34.051 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:34.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:34Z" } 2023-09-29T22:52:34.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:34.052 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:34.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:34.053 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:52:39.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:39.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:39.055 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:39.103 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:39.103 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:39.150 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:39.149+0000 7f57cea19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:39.151 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:39.150+0000 7f57cea19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:39.164 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:39.164 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:39.164 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:39.164 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:39.165 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:39.167 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.242855_235.1", 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.242855Z", 2023-09-29T22:52:39.168 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:39.169 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:39.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:39.171 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:39Z" 2023-09-29T22:52:39.171 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:39.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.242855_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.242855Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:39Z" } 2023-09-29T22:52:39.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.242855_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.242855Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:39.172 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:39.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.242855_235.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:39.172 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.242855_235.1 2023-09-29T22:52:44.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:44.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:44.173 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:44.219 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:44.220 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:44.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:44.258+0000 7f3c33c17880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:44.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:44.259+0000 7f3c33c17880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:44.274 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:44.274 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:44.275 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.276 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:44.276 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:44.276 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:44.276 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:44.276 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:44.277 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:44.278 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:44.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:44.278 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:44.278 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:44.279 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:44.280 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:44.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:44.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:44.280 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026484.251433_236.1", 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:44.281 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:44.282 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:44.283 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:44.283 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:44.283 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:44.283 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:44.283 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:44.284 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:44.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.285 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:44.285 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:44.285 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:44.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:44.288 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:44Z" 2023-09-29T22:52:44.288 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:44.289 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026484.251433_236.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:04.251433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:44Z" } 2023-09-29T22:52:44.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.251433_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.251433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:44.290 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:44.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.251433_236.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:44.290 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.251433_236.1 2023-09-29T22:52:49.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:49.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:49.291 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:49.339 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:49.339 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:49.373 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:49.373+0000 7f22e5c0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:49.374 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:49.374+0000 7f22e5c0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:49.387 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:49.387 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:49.387 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.388 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:49.389 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:49.390 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:49.390 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:49.390 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:49.390 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:49.390 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:49.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:49.391 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:49.391 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:49.392 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:49.392 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:49.392 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:49.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:49.392 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:49.393 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:49.393 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:49.393 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:49.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.111037_224.1", 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:49.394 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:49.395 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:49.395 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-29T22:52:49.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:49.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:49.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:49.396 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:49.396 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:49.396 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:49.396 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:49.397 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:49.397 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:49.397 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:49.397 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:49.397 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:49.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:49.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:49.398 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:49.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.398 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:49.399 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:49.399 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:49.399 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:49.399 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:49Z" 2023-09-29T22:52:49.399 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:49.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.111037_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.111037Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:49Z" } 2023-09-29T22:52:49.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.111037_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.111037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:49.401 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:49.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.111037_224.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:49.402 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.111037_224.1 2023-09-29T22:52:54.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:54.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:54.403 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:54.456 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:54.456 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:54.498 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:54.499+0000 7f22e1927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:54.499 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:54.500+0000 7f22e1927880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:54.557 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:54.558 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.559 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:54.560 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:54.561 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.562 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:54.563 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:54.564 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:54.565 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:54Z" 2023-09-29T22:52:54.565 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:54.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:54Z" } 2023-09-29T22:52:54.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:54.566 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:54.566 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:54.566 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:52:59.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:59.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:52:59.568 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:52:59.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:52:59.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:52:59.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:59.647+0000 7fbd3db29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:59.647 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:52:59.647+0000 7fbd3db29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:52:59.661 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:52:59.662 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:52:59.663 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:59.664 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:59.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.127019_228.1", 2023-09-29T22:52:59.666 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.127019Z", 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:59.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:52:59.668 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:52:59.669 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:52:59.669 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:52:59.669 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:52:59.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:59.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:52:59.670 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:52:59Z" 2023-09-29T22:52:59.671 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:52:59.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.127019_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.127019Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:52:59Z" } 2023-09-29T22:52:59.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.127019_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.127019Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:52:59.672 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:59.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.127019_228.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:52:59.673 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.127019_228.1 2023-09-29T22:53:04.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:04.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:04.675 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:04.727 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:04.727 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:04.763 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:04.763+0000 7f804e733880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:04.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:04.764+0000 7f804e733880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:04.781 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.201655_230.1", 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.201655Z", 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:04.786 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:04Z" 2023-09-29T22:53:04.787 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:04.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.201655_230.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.201655Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:04Z" } 2023-09-29T22:53:04.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.201655_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.201655Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:04.789 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:04.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.201655_230.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:04.789 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.201655_230.1 2023-09-29T22:53:09.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:09.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:09.790 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:09.839 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:09.839 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:09.871+0000 7f7d22b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:09.872+0000 7f7d22b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:09.886 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:09.886 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:09.886 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:09.886 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:09.887 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:09.888 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.889 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.079039_218.1", 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:09.890 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:09.891 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:09.892 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:09Z" 2023-09-29T22:53:09.893 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:09.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.079039_218.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.079039Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:09Z" } 2023-09-29T22:53:09.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:09.894 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:09.894 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:09.895 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-29T22:53:14.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:14.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:14.896 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:14.945 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:14.946 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:14.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:14.996+0000 7f395a529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:14.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:14.997+0000 7f395a529880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:15.018 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:15.018 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:15.019 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:15.020 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:15.021 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:53:15.022 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:15.023 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:15.024 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:15.025 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:15.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:15.025 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:14Z" 2023-09-29T22:53:15.025 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:15.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:14Z" } 2023-09-29T22:53:15.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:15.026 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:15.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:15.026 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:53:20.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:20.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:20.028 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:20.082 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:20.082 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:20.125 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:20.125+0000 7ff2f1019880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:20.126 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:20.126+0000 7ff2f1019880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:20.144 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:20.145 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:20.146 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:20.147 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:20.148 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:20.149 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:20Z" 2023-09-29T22:53:20.150 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:20.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:20Z" } 2023-09-29T22:53:20.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:20.151 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:20.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:20.152 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:53:25.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:25.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:25.153 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:25.201 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:25.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:25.252+0000 7f63114b6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:25.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:25.253+0000 7f63114b6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:25.275 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.276 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:25.277 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:25.278 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:25.279 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:25.280 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.281 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:25.282 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:25.282 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:25.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:25.282 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:25Z" 2023-09-29T22:53:25.282 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:25.283 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:25Z" } 2023-09-29T22:53:25.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:25.283 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:25.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:25.283 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:53:30.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:30.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:30.285 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:30.336 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:30.336 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:30.371 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:30.371+0000 7fcda7e13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:30.372 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:30.372+0000 7fcda7e13880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:30.389 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:30.389 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:30.389 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:30.389 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:30.390 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.391 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:30.392 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:30.393 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.265680_238.1", 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.265680Z", 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:30.394 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:30.395 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:30Z" 2023-09-29T22:53:30.396 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:30.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.265680_238.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.265680Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:30Z" } 2023-09-29T22:53:30.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.265680_238.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.265680Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:30.397 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:30.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.265680_238.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:30.398 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.265680_238.1 2023-09-29T22:53:35.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:35.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:35.399 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:35.454 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:35.455 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:35.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:35.489+0000 7f9595333880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:35.490 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:35.490+0000 7f9595333880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:35.506 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:35.507 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:35.508 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:35.509 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.113101_225.1", 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:35.511 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:35Z" 2023-09-29T22:53:35.512 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:35.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.113101_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.113101Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:35Z" } 2023-09-29T22:53:35.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.113101_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.113101Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:35.514 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:35.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.113101_225.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:35.514 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.113101_225.1 2023-09-29T22:53:40.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:40.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:40.515 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:40.562 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:40.562 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:40.603 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:40.602+0000 7f4ea5fdc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:40.604 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:40.603+0000 7f4ea5fdc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:40.619 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:40.620 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:40.621 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:40.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:40.623 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:40.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:40Z" 2023-09-29T22:53:40.625 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:40.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:40Z" } 2023-09-29T22:53:40.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:40.626 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:40.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:40.627 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:53:45.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:45.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:45.628 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:45.678 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:45.678 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:45.719 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:45.719+0000 7f0dab52c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:45.720 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:45.720+0000 7f0dab52c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:45.736 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:45.736 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:45.736 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:45.736 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:45.737 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:45.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:45.739 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:45.740 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:45.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:45.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:45Z" 2023-09-29T22:53:45.743 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:45.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:45Z" } 2023-09-29T22:53:45.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:45.745 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:45.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:45.745 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:53:50.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:50.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:50.746 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:50.801 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:50.802 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:50.837 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:50.837+0000 7f6291e44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:50.838 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:50.838+0000 7f6291e44880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:50.853 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:50.853 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:50.853 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:50.854 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:50.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:50.856 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.233377_234.1", 2023-09-29T22:53:50.857 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.233377Z", 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:50.858 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:50.859 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:50Z" 2023-09-29T22:53:50.860 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:50.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.233377_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.233377Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:50Z" } 2023-09-29T22:53:50.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.233377_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.233377Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:50.862 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:50.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.233377_234.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:50.862 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.233377_234.1 2023-09-29T22:53:55.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:55.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:53:55.864 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:53:55.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:53:55.910 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:53:55.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:55.944+0000 7f6d2d326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:55.944 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:53:55.944+0000 7f6d2d326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:53:55.958 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:53:55.958 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:53:55.958 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:53:55.958 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:55.959 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:53:55.960 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:55.962 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:53:55.963 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:53:55Z" 2023-09-29T22:53:55.964 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:53:55.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:53:55Z" } 2023-09-29T22:53:55.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:53:55.966 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:55.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:53:55.966 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:54:00.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:00.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:00.967 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:01.018 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:01.018 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:01.064 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:01.064+0000 7f53c5326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:01.065 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:01.065+0000 7f53c5326880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:01.082 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:01.083 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:01.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:01.085 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:01.086 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:01.087 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:01.088 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:01.089 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:01.089 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:01Z" 2023-09-29T22:54:01.089 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:01.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:01Z" } 2023-09-29T22:54:01.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:01.090 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:01.090 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:01.090 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:54:06.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:06.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:06.091 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:06.159 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:06.159 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:06.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:06.270+0000 7fad3ac8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:06.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:06.271+0000 7fad3ac8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:06.288 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:06.289 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:06.290 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:06.291 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:06.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:06.293 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:06.294 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:06Z" 2023-09-29T22:54:06.295 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:06.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:06Z" } 2023-09-29T22:54:06.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:06.297 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:06.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:06.297 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:54:11.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:11.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:11.298 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:11.348 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:11.348 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:11.396 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:11.396+0000 7f379de67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:11.397 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:11.397+0000 7f379de67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:11.415 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:11.415 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:11.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:11.417 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:11.418 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:11.419 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.095615_221.1", 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:11Z" 2023-09-29T22:54:11.422 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:11.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.095615_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.095615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:11Z" } 2023-09-29T22:54:11.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:11.424 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:11.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:11.424 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-29T22:54:16.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:16.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:16.426 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:16.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:16.478 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:16.515 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:16.515+0000 7f472aeb6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:16.516 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:16.516+0000 7f472aeb6880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:16.534 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.535 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:16.536 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:16.537 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:16.538 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.093360_220.1", 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:16.540 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:16Z" 2023-09-29T22:54:16.541 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:16.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.093360_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.093360Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:16Z" } 2023-09-29T22:54:16.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:16.543 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:16.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:16.543 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-29T22:54:21.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:21.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:21.544 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:21.623 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:21.623 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:21.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:21.655+0000 7f499c11f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:21.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:21.656+0000 7f499c11f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:21.670 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:21.670 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:21.670 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:21.670 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:21.670 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:21.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:21.672 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:21.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:21.674 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:21.675 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:21.676 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:21.677 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:21Z" 2023-09-29T22:54:21.678 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:21.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:21Z" } 2023-09-29T22:54:21.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:21.679 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:21.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:21.680 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:54:26.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:26.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:26.681 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:26.730 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:26.730 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:26.775 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:26.775+0000 7efd6832d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:26.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:26.776+0000 7efd6832d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:26.795 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:26.795 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:26.795 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:26.796 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:26.797 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:26.798 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:26.800 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:26.801 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:26.802 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:26Z" 2023-09-29T22:54:26.803 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:26.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:26Z" } 2023-09-29T22:54:26.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:26.805 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:26.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:26.805 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:54:31.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:31.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:31.807 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:31.859 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:31.860 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:31.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:31.895+0000 7effcc69d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:31.896 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:31.896+0000 7effcc69d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:31.911 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:31.911 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:31.911 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:31.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:31.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.233377_234.1", 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.233377Z", 2023-09-29T22:54:31.915 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:31.916 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:31.917 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:31.919 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:31.919 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:31Z" 2023-09-29T22:54:31.919 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:31.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.233377_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.233377Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:31Z" } 2023-09-29T22:54:31.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.233377_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.233377Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:31.920 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:31.920 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.233377_234.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:31.920 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.233377_234.1 2023-09-29T22:54:36.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:36.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:36.922 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:36.990 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:36.991 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:37.041 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:37.040+0000 7fd49a40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:37.042 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:37.041+0000 7fd49a40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:37.061 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:37.062 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:37.063 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:37.064 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:37.065 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.123832_226.1", 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:37.066 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:37.067 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:37Z" 2023-09-29T22:54:37.068 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:37.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.123832_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.123832Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:37Z" } 2023-09-29T22:54:37.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:37.070 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:37.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:37.070 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-29T22:54:42.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:42.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:42.072 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:42.124 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:42.124 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:42.175 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:42.174+0000 7f1c91125880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:42.176 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:42.175+0000 7f1c91125880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:42.198 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:42.199 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:42.200 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:42.201 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.081380_219.1", 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:42.202 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:42.203 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:42Z" 2023-09-29T22:54:42.204 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:42.205 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.081380_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.081380Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:42Z" } 2023-09-29T22:54:42.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:42.205 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:42.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:42.206 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-29T22:54:47.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:47.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:47.207 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:47.260 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:47.261 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:47.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:47.298+0000 7f1427f29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:47.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:47.299+0000 7f1427f29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:47.324 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:47.325 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:47.326 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:47.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026485.281741_240.1", 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:47.328 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:05.281741Z", 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:47.329 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:47.330 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:47Z" 2023-09-29T22:54:47.331 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:47.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026485.281741_240.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:05.281741Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:47Z" } 2023-09-29T22:54:47.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.281741_240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.281741Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:47.333 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:47.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.281741_240.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:47.333 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026485.281741_240.1 2023-09-29T22:54:52.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:52.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:52.334 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:52.383 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:52.384 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:52.431+0000 7fb6b2122880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:52.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:52.432+0000 7fb6b2122880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:52.447 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:52.447 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:52.447 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:52.447 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:52.447 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:52.448 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:52.449 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:52.450 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696026483.026351_217.1", 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:52.452 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:52.453 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:52.454 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:52.454 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:52.454 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:52Z" 2023-09-29T22:54:52.454 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:52.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696026483.026351_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:28:03.026351Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:52Z" } 2023-09-29T22:54:52.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:52.455 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:52.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:52.455 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-29T22:54:57.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:57.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:54:57.457 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:54:57.510 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:57.510 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:57.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:57.559+0000 7fdc7231c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:57.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:54:57.560+0000 7fdc7231c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:54:57.575 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:54:57.575 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:57.576 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:57.577 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:57.578 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:54:57.579 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:54:57.580 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:54:57.581 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:54:57Z" 2023-09-29T22:54:57.582 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:54:57.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:54:57Z" } 2023-09-29T22:54:57.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:54:57.584 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:57.584 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:54:57.584 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:54:57.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:54:57.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXNE2CFXHY4T6VO3', '--secret', '0At0HHaplw6uG1twoVspt6vkyBiGHNxa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:54:57.585 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HXNE2CFXHY4T6VO3 --secret 0At0HHaplw6uG1twoVspt6vkyBiGHNxa --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:54:57.640 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:57.640 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:57.657 INFO:teuthology.orchestra.run.smithi184.stderr:NOTICE: set zone's realm_id=de44a8c3-8a59-46e4-824e-eaf8e5fcea33 2023-09-29T22:54:57.672 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:54:57.672 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "68f823da-b479-485a-a3eb-1caa585b801f", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "remove.rgw.meta:root", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "remove.rgw.control", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "remove.rgw.log:gc", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "remove.rgw.log:lc", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "remove.rgw.log", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-09-29T22:54:57.673 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "remove.rgw.otp", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:54:57.674 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "remove.rgw.buckets.index", 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "remove.rgw.buckets.data" 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-09-29T22:54:57.675 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0, 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: "inline_data": true 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "remove.rgw.log:notif" 2023-09-29T22:54:57.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:54:57.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:54:57.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:54:57.678 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:54:57.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:54:57.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:54:57.737 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:54:57.738+0000 7f7c5c327880 0 period (6a43d71a-d207-46f3-bdd8-e7a78954a184 does not have zone 68f823da-b479-485a-a3eb-1caa585b801f configured 2023-09-29T22:55:04.358 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 5, 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2023-09-29T22:55:04.389 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": true, 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:55:04.390 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:55:04.391 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:55:04.392 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "68f823da-b479-485a-a3eb-1caa585b801f", 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:55:04.393 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:55:04.394 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:55:04.395 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": true, 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": false, 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": true, 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.397 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": false, 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": false, 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "", 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "supported_features": [ 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "compress-encrypted", 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:04.398 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2023-09-29T22:55:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2023-09-29T22:55:04.399 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2023-09-29T22:55:04.401 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2023-09-29T22:55:04.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled_features": [ 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: "resharding" 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2885950644 2023-09-29T22:55:04.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "68f823da-b479-485a-a3eb-1caa585b801f", 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 4254555811 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2464975307 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:04.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1018467004 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:55:04.405 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2023-09-29T22:55:04.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "user_ratelimit": { 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_ratelimit": { 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.407 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "anonymous_ratelimit": { 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.408 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:04.409 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:04.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.411 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:04.459 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.460 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:04.478 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.479 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.480 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:04.481 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.482 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.483 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.485 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.486 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:55:04.487 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:04.488 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.489 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.489 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:04.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.490 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:04.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:04.553 INFO:teuthology.orchestra.run.smithi175.stderr:zone name remove not found in zonegroup test-zonegroup 2023-09-29T22:55:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-09-29T22:55:04.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.558 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:04.617 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.617 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:04.650 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:04.649+0000 7fe291067880 0 Removed zone from zonegroup test-zonegroup 2023-09-29T22:55:04.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.658 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:04.719 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.719 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:04.752 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:55:04.752+0000 7f8dcbd2d880 0 WARNING: can't generate connection for zone 68f823da-b479-485a-a3eb-1caa585b801f id remove: no endpoints defined 2023-09-29T22:55:04.877 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 6, 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-29T22:55:04.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:04.879 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.880 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.881 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.882 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:55:04.883 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.883 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.883 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.884 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-29T22:55:04.885 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:04.886 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.887 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.888 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:04.889 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.890 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.891 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-29T22:55:04.926 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.926 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:55:04.927 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:04.928 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:04.929 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:04.930 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-29T22:55:04.931 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:55:04.932 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:04.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-09-29T22:55:04.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:04.934 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:04.989 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.989 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.028 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "c4b6c93f-57da-477d-955a-ff39bb98f98f", 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove_zg", 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "remove_zg", 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": false, 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:05.029 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "", 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [], 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [] 2023-09-29T22:55:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:05.032 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.032 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.032 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:05.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.034 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.086 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.087 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.128 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:55:05.129+0000 7fed2f12a880 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-09-29T22:55:05.134 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:05.134 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "7e94b72d-2b81-4397-b24c-95f61eda76a2", 2023-09-29T22:55:05.134 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove", 2023-09-29T22:55:05.134 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "remove.rgw.meta:root", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "remove.rgw.control", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "remove.rgw.log:gc", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "remove.rgw.log:lc", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "remove.rgw.log", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-09-29T22:55:05.135 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "remove.rgw.otp", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2023-09-29T22:55:05.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "remove.rgw.buckets.index", 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "remove.rgw.buckets.data" 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0, 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: "inline_data": true 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.138 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.139 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "remove.rgw.log:notif" 2023-09-29T22:55:05.139 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:05.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.140 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.190 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.191 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.280 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 7, 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.281 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "c4b6c93f-57da-477d-955a-ff39bb98f98f", 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove_zg", 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "remove_zg", 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": false, 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "7e94b72d-2b81-4397-b24c-95f61eda76a2", 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:55:05.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "7e94b72d-2b81-4397-b24c-95f61eda76a2", 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove", 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": false, 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.283 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.284 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:05.284 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.284 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.284 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.284 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.285 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.285 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.285 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.286 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:05.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.286 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:05.286 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:05.287 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:55:05.287 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:55:05.287 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.287 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.288 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:05.288 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.288 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:05.288 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:05.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.289 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:05.289 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.289 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.290 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:55:05.290 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:55:05.290 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:55:05.290 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:55:05.291 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.291 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.291 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.291 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:05.292 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:05.292 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.292 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:55:05.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.292 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:05.293 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:55:05.293 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.293 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:55:05.293 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.293 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.294 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.294 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.294 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.294 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.295 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:05.295 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.295 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.295 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.295 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.296 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:55:05.297 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.297 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.297 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.297 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.297 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.298 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.298 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.299 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.306 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.307 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:05.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:05.309 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.310 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "7e94b72d-2b81-4397-b24c-95f61eda76a2", 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 15566784 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-29T22:55:05.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.312 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:05.313 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.314 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-29T22:55:05.315 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:55:05.316 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:05.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.318 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.368 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.369 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.395 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:55:05.395+0000 7f6692a14880 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-09-29T22:55:05.399 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:55:05.399+0000 7f6692a14880 0 Removed zone from zonegroup remove_zg 2023-09-29T22:55:05.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.413 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.481 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.482 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.509 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:55:05.509+0000 7fad59012880 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-09-29T22:55:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-09-29T22:55:05.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.517 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.568 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:05.595 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:05.645 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.645 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.783 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 8, 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-29T22:55:05.784 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-29T22:55:05.785 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-29T22:55:05.786 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.787 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.788 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.789 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.790 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.791 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-29T22:55:05.792 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-29T22:55:05.793 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-29T22:55:05.794 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.795 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-29T22:55:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.797 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-29T22:55:05.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:05.836 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:55:05.836 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-29T22:55:05.836 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-29T22:55:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-29T22:55:05.838 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.839 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.840 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-29T22:55:05.841 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-29T22:55:05.842 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-29T22:55:05.842 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-29T22:55:05.842 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:55:05.843 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:55:05.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-09-29T22:55:05.845 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-46 2023-09-29T22:55:05.845 DEBUG:boto:path=/dmuhsu-46/ 2023-09-29T22:55:05.845 DEBUG:boto:auth_path=/dmuhsu-46/ 2023-09-29T22:55:05.845 DEBUG:boto:Method: PUT 2023-09-29T22:55:05.846 DEBUG:boto:Path: /dmuhsu-46/ 2023-09-29T22:55:05.846 DEBUG:boto:Data: 2023-09-29T22:55:05.846 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.846 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:05.846 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.846 DEBUG:boto:Params: {} 2023-09-29T22:55:05.846 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:55:05.847 DEBUG:boto:Token: None 2023-09-29T22:55:05.847 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:05 GMT /dmuhsu-46/ 2023-09-29T22:55:05.847 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lAZj1Szo79cR8MjmYWmDtybgsBQ= 2023-09-29T22:55:05.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lAZj1Szo79cR8MjmYWmDtybgsBQ=', 'Content-Length': '0'} 2023-09-29T22:55:06.659 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e73eac63ea6e8c5-00651755ca-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.659 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-47 2023-09-29T22:55:06.659 DEBUG:boto:path=/dmuhsu-47/ 2023-09-29T22:55:06.660 DEBUG:boto:auth_path=/dmuhsu-47/ 2023-09-29T22:55:06.660 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.660 DEBUG:boto:Path: /dmuhsu-47/ 2023-09-29T22:55:06.660 DEBUG:boto:Data: 2023-09-29T22:55:06.660 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.660 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:55:06.660 DEBUG:boto:Port: 8000 2023-09-29T22:55:06.660 DEBUG:boto:Params: {} 2023-09-29T22:55:06.660 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:55:06.661 DEBUG:boto:Token: None 2023-09-29T22:55:06.661 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-47/ 2023-09-29T22:55:06.661 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:09h6fm15eEsh+/r63S7U/MovTGY= 2023-09-29T22:55:06.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:09h6fm15eEsh+/r63S7U/MovTGY=', 'Content-Length': '0'} 2023-09-29T22:55:06.806 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016ff358ebe2e0193-00651755ca-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.806 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-48 2023-09-29T22:55:06.807 DEBUG:boto:path=/dmuhsu-48/ 2023-09-29T22:55:06.807 DEBUG:boto:auth_path=/dmuhsu-48/ 2023-09-29T22:55:06.807 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.807 DEBUG:boto:Path: /dmuhsu-48/ 2023-09-29T22:55:06.807 DEBUG:boto:Data: 2023-09-29T22:55:06.807 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.807 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:55:06.807 DEBUG:boto:Port: 8001 2023-09-29T22:55:06.807 DEBUG:boto:Params: {} 2023-09-29T22:55:06.808 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:55:06.808 DEBUG:boto:Token: None 2023-09-29T22:55:06.808 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-48/ 2023-09-29T22:55:06.808 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eiIw8XDn3JKFH5hgA98TciAByDI= 2023-09-29T22:55:06.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eiIw8XDn3JKFH5hgA98TciAByDI=', 'Content-Length': '0'} 2023-09-29T22:55:06.884 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005af96d0d6ee8d250-00651755ca-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.884 DEBUG:boto:path=/dmuhsu-46/ 2023-09-29T22:55:06.884 DEBUG:boto:auth_path=/dmuhsu-46/ 2023-09-29T22:55:06.884 DEBUG:boto:path=/dmuhsu-46/?website 2023-09-29T22:55:06.884 DEBUG:boto:auth_path=/dmuhsu-46/?website 2023-09-29T22:55:06.884 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.885 DEBUG:boto:Path: /dmuhsu-46/?website 2023-09-29T22:55:06.885 DEBUG:boto:Data: index.htmlerror.html 2023-09-29T22:55:06.885 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.885 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:06.885 DEBUG:boto:Port: 8000 2023-09-29T22:55:06.885 DEBUG:boto:Params: {} 2023-09-29T22:55:06.885 DEBUG:boto:Token: None 2023-09-29T22:55:06.885 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-46/?website 2023-09-29T22:55:06.886 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Sil2SBwGjygYMqfCQLFAoa8fbpI= 2023-09-29T22:55:06.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Sil2SBwGjygYMqfCQLFAoa8fbpI=', 'Content-Length': '241'} 2023-09-29T22:55:06.886 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000fbae396467bed8d8-00651755ca-1e9c-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-09-29T22:55:06.888 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-49 2023-09-29T22:55:06.888 DEBUG:boto:path=/dmuhsu-49/ 2023-09-29T22:55:06.888 DEBUG:boto:auth_path=/dmuhsu-49/ 2023-09-29T22:55:06.888 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.888 DEBUG:boto:Path: /dmuhsu-49/ 2023-09-29T22:55:06.888 DEBUG:boto:Data: 2023-09-29T22:55:06.888 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.889 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:06.889 DEBUG:boto:Port: 8000 2023-09-29T22:55:06.889 DEBUG:boto:Params: {} 2023-09-29T22:55:06.889 DEBUG:boto:Token: None 2023-09-29T22:55:06.889 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-49/ 2023-09-29T22:55:06.889 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:j5rU3x4x7czS+u1VB7QlK6yhLtI= 2023-09-29T22:55:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j5rU3x4x7czS+u1VB7QlK6yhLtI=', 'Content-Length': '0'} 2023-09-29T22:55:06.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad81a67f7fb01e84-00651755ca-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.932 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-50 2023-09-29T22:55:06.932 DEBUG:boto:path=/dmuhsu-50/ 2023-09-29T22:55:06.932 DEBUG:boto:auth_path=/dmuhsu-50/ 2023-09-29T22:55:06.932 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.932 DEBUG:boto:Path: /dmuhsu-50/ 2023-09-29T22:55:06.933 DEBUG:boto:Data: 2023-09-29T22:55:06.933 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.933 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:55:06.933 DEBUG:boto:Port: 8000 2023-09-29T22:55:06.933 DEBUG:boto:Params: {} 2023-09-29T22:55:06.933 DEBUG:boto:Token: None 2023-09-29T22:55:06.933 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-50/ 2023-09-29T22:55:06.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+P6e8vjx66Q0t6tVC6F3TU0VZbo= 2023-09-29T22:55:06.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+P6e8vjx66Q0t6tVC6F3TU0VZbo=', 'Content-Length': '0'} 2023-09-29T22:55:06.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9e4039f49c62469-00651755ca-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.965 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-51 2023-09-29T22:55:06.966 DEBUG:boto:path=/dmuhsu-51/ 2023-09-29T22:55:06.966 DEBUG:boto:auth_path=/dmuhsu-51/ 2023-09-29T22:55:06.966 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.966 DEBUG:boto:Path: /dmuhsu-51/ 2023-09-29T22:55:06.966 DEBUG:boto:Data: 2023-09-29T22:55:06.967 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.967 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:55:06.967 DEBUG:boto:Port: 8001 2023-09-29T22:55:06.967 DEBUG:boto:Params: {} 2023-09-29T22:55:06.967 DEBUG:boto:Token: None 2023-09-29T22:55:06.968 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /dmuhsu-51/ 2023-09-29T22:55:06.968 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+1yPevUkFNITn92TPzGryip8tG0= 2023-09-29T22:55:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+1yPevUkFNITn92TPzGryip8tG0=', 'Content-Length': '0'} 2023-09-29T22:55:07.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002852c31901c2dce2-00651755ca-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:07.044 DEBUG:boto:path=/dmuhsu-49/ 2023-09-29T22:55:07.044 DEBUG:boto:auth_path=/dmuhsu-49/ 2023-09-29T22:55:07.044 DEBUG:boto:path=/dmuhsu-49/?policy 2023-09-29T22:55:07.045 DEBUG:boto:auth_path=/dmuhsu-49/?policy 2023-09-29T22:55:07.045 DEBUG:boto:Method: PUT 2023-09-29T22:55:07.045 DEBUG:boto:Path: /dmuhsu-49/?policy 2023-09-29T22:55:07.045 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:55:07.045 DEBUG:boto:Headers: {} 2023-09-29T22:55:07.045 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:07.045 DEBUG:boto:Port: 8000 2023-09-29T22:55:07.045 DEBUG:boto:Params: {} 2023-09-29T22:55:07.046 DEBUG:boto:Token: None 2023-09-29T22:55:07.046 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:07 GMT /dmuhsu-49/?policy 2023-09-29T22:55:07.046 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7FDfB75GkzUcTspYH2v4GzzKI3k= 2023-09-29T22:55:07.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7FDfB75GkzUcTspYH2v4GzzKI3k=', 'Content-Length': '97'} 2023-09-29T22:55:07.059 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b675dfdf0fa49bce-00651755cb-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:55:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:07.059 DEBUG:boto:path=/dmuhsu-49/ 2023-09-29T22:55:07.059 DEBUG:boto:auth_path=/dmuhsu-49/ 2023-09-29T22:55:07.059 DEBUG:boto:path=/dmuhsu-49/?policy 2023-09-29T22:55:07.059 DEBUG:boto:auth_path=/dmuhsu-49/?policy 2023-09-29T22:55:07.059 DEBUG:boto:Method: GET 2023-09-29T22:55:07.060 DEBUG:boto:Path: /dmuhsu-49/?policy 2023-09-29T22:55:07.060 DEBUG:boto:Data: 2023-09-29T22:55:07.060 DEBUG:boto:Headers: {} 2023-09-29T22:55:07.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:07.060 DEBUG:boto:Port: 8000 2023-09-29T22:55:07.060 DEBUG:boto:Params: {} 2023-09-29T22:55:07.060 DEBUG:boto:Token: None 2023-09-29T22:55:07.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:07 GMT /dmuhsu-49/?policy 2023-09-29T22:55:07.061 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wl9xqhqhd9IqR8a3r2ieNEGpJnY= 2023-09-29T22:55:07.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wl9xqhqhd9IqR8a3r2ieNEGpJnY=', 'Content-Length': '0'} 2023-09-29T22:55:07.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e478ac4ccb10b019-00651755cb-1e9c-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:55:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:07.062 DEBUG:boto:path=/dmuhsu-50/ 2023-09-29T22:55:07.062 DEBUG:boto:auth_path=/dmuhsu-50/ 2023-09-29T22:55:07.062 DEBUG:boto:path=/dmuhsu-50/?policy 2023-09-29T22:55:07.062 DEBUG:boto:auth_path=/dmuhsu-50/?policy 2023-09-29T22:55:07.063 DEBUG:boto:Method: PUT 2023-09-29T22:55:07.063 DEBUG:boto:Path: /dmuhsu-50/?policy 2023-09-29T22:55:07.063 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:55:07.063 DEBUG:boto:Headers: {} 2023-09-29T22:55:07.063 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:55:07.063 DEBUG:boto:Port: 8000 2023-09-29T22:55:07.063 DEBUG:boto:Params: {} 2023-09-29T22:55:07.063 DEBUG:boto:Token: None 2023-09-29T22:55:07.064 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:07 GMT /dmuhsu-50/?policy 2023-09-29T22:55:07.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JDgLgjmShkGvLzlqstPKkyF3nQc= 2023-09-29T22:55:07.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:07 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JDgLgjmShkGvLzlqstPKkyF3nQc=', 'Content-Length': '97'} 2023-09-29T22:55:08.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ccb42ebb6002d729-00651755cb-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:55:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:08.075 DEBUG:boto:path=/dmuhsu-50/ 2023-09-29T22:55:08.075 DEBUG:boto:auth_path=/dmuhsu-50/ 2023-09-29T22:55:08.075 DEBUG:boto:path=/dmuhsu-50/?policy 2023-09-29T22:55:08.075 DEBUG:boto:auth_path=/dmuhsu-50/?policy 2023-09-29T22:55:08.075 DEBUG:boto:Method: GET 2023-09-29T22:55:08.075 DEBUG:boto:Path: /dmuhsu-50/?policy 2023-09-29T22:55:08.076 DEBUG:boto:Data: 2023-09-29T22:55:08.076 DEBUG:boto:Headers: {} 2023-09-29T22:55:08.076 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:55:08.076 DEBUG:boto:Port: 8000 2023-09-29T22:55:08.076 DEBUG:boto:Params: {} 2023-09-29T22:55:08.076 DEBUG:boto:Token: None 2023-09-29T22:55:08.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:08 GMT /dmuhsu-50/?policy 2023-09-29T22:55:08.076 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vy7JSFPDQLT1/jJZaIZGqesU6MY= 2023-09-29T22:55:08.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vy7JSFPDQLT1/jJZaIZGqesU6MY=', 'Content-Length': '0'} 2023-09-29T22:55:08.078 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001bd2256693e2e2f6-00651755cc-14ab-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:55:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:08.078 DEBUG:boto:path=/dmuhsu-51/ 2023-09-29T22:55:08.078 DEBUG:boto:auth_path=/dmuhsu-51/ 2023-09-29T22:55:08.078 DEBUG:boto:path=/dmuhsu-51/?policy 2023-09-29T22:55:08.079 DEBUG:boto:auth_path=/dmuhsu-51/?policy 2023-09-29T22:55:08.079 DEBUG:boto:Method: PUT 2023-09-29T22:55:08.079 DEBUG:boto:Path: /dmuhsu-51/?policy 2023-09-29T22:55:08.079 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:55:08.080 DEBUG:boto:Headers: {} 2023-09-29T22:55:08.080 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:55:08.080 DEBUG:boto:Port: 8001 2023-09-29T22:55:08.080 DEBUG:boto:Params: {} 2023-09-29T22:55:08.081 DEBUG:boto:Token: None 2023-09-29T22:55:08.081 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:08 GMT /dmuhsu-51/?policy 2023-09-29T22:55:08.081 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CLMogaoqzFdbXI1qBcpx9SH8Rs4= 2023-09-29T22:55:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CLMogaoqzFdbXI1qBcpx9SH8Rs4=', 'Content-Length': '97'} 2023-09-29T22:55:09.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011acb0637a2e3b0a-00651755cc-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:09.091 DEBUG:boto:path=/dmuhsu-51/ 2023-09-29T22:55:09.092 DEBUG:boto:auth_path=/dmuhsu-51/ 2023-09-29T22:55:09.092 DEBUG:boto:path=/dmuhsu-51/?policy 2023-09-29T22:55:09.092 DEBUG:boto:auth_path=/dmuhsu-51/?policy 2023-09-29T22:55:09.092 DEBUG:boto:Method: GET 2023-09-29T22:55:09.092 DEBUG:boto:Path: /dmuhsu-51/?policy 2023-09-29T22:55:09.092 DEBUG:boto:Data: 2023-09-29T22:55:09.092 DEBUG:boto:Headers: {} 2023-09-29T22:55:09.092 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:55:09.092 DEBUG:boto:Port: 8001 2023-09-29T22:55:09.092 DEBUG:boto:Params: {} 2023-09-29T22:55:09.093 DEBUG:boto:Token: None 2023-09-29T22:55:09.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:09 GMT /dmuhsu-51/?policy 2023-09-29T22:55:09.093 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2I0gU4gDTg8lRRQFJDBSiRJsEAQ= 2023-09-29T22:55:09.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2I0gU4gDTg8lRRQFJDBSiRJsEAQ=', 'Content-Length': '0'} 2023-09-29T22:55:09.094 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b39dc26da62ec08-00651755cd-1ea0-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:09.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-09-29T22:55:09.095 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-52 2023-09-29T22:55:09.095 DEBUG:boto:path=/dmuhsu-52/ 2023-09-29T22:55:09.095 DEBUG:boto:auth_path=/dmuhsu-52/ 2023-09-29T22:55:09.095 DEBUG:boto:Method: PUT 2023-09-29T22:55:09.096 DEBUG:boto:Path: /dmuhsu-52/ 2023-09-29T22:55:09.096 DEBUG:boto:Data: 2023-09-29T22:55:09.096 DEBUG:boto:Headers: {} 2023-09-29T22:55:09.096 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T22:55:09.096 DEBUG:boto:Port: 8000 2023-09-29T22:55:09.096 DEBUG:boto:Params: {} 2023-09-29T22:55:09.096 DEBUG:boto:Token: None 2023-09-29T22:55:09.096 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:09 GMT /dmuhsu-52/ 2023-09-29T22:55:09.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ggwFeLIc4npdYMUhMUsLeLjJwEc= 2023-09-29T22:55:09.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ggwFeLIc4npdYMUhMUsLeLjJwEc=', 'Content-Length': '0'} 2023-09-29T22:55:09.114 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ea01ca120eef615e-00651755cd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:09.114 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-53 2023-09-29T22:55:09.114 DEBUG:boto:path=/dmuhsu-53/ 2023-09-29T22:55:09.114 DEBUG:boto:auth_path=/dmuhsu-53/ 2023-09-29T22:55:09.115 DEBUG:boto:Method: PUT 2023-09-29T22:55:09.115 DEBUG:boto:Path: /dmuhsu-53/ 2023-09-29T22:55:09.115 DEBUG:boto:Data: 2023-09-29T22:55:09.115 DEBUG:boto:Headers: {} 2023-09-29T22:55:09.115 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T22:55:09.115 DEBUG:boto:Port: 8000 2023-09-29T22:55:09.115 DEBUG:boto:Params: {} 2023-09-29T22:55:09.115 DEBUG:boto:Token: None 2023-09-29T22:55:09.116 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:09 GMT /dmuhsu-53/ 2023-09-29T22:55:09.116 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E9AKkpBtj7R1N4lWigBjuoddVV4= 2023-09-29T22:55:09.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E9AKkpBtj7R1N4lWigBjuoddVV4=', 'Content-Length': '0'} 2023-09-29T22:55:09.145 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e049104c38943f3-00651755cd-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:09.145 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-54 2023-09-29T22:55:09.146 DEBUG:boto:path=/dmuhsu-54/ 2023-09-29T22:55:09.146 DEBUG:boto:auth_path=/dmuhsu-54/ 2023-09-29T22:55:09.146 DEBUG:boto:Method: PUT 2023-09-29T22:55:09.146 DEBUG:boto:Path: /dmuhsu-54/ 2023-09-29T22:55:09.146 DEBUG:boto:Data: 2023-09-29T22:55:09.146 DEBUG:boto:Headers: {} 2023-09-29T22:55:09.146 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T22:55:09.146 DEBUG:boto:Port: 8001 2023-09-29T22:55:09.146 DEBUG:boto:Params: {} 2023-09-29T22:55:09.147 DEBUG:boto:Token: None 2023-09-29T22:55:09.147 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:09 GMT /dmuhsu-54/ 2023-09-29T22:55:09.147 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MTrP4+ToJCvF4or7JhV87q51z6A= 2023-09-29T22:55:09.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MTrP4+ToJCvF4or7JhV87q51z6A=', 'Content-Length': '0'} 2023-09-29T22:55:09.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bb155adc81ade653-00651755cd-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:09.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:09.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:09.181 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:09.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:09.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:09.426 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T22:55:09.442 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:09.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028109.165504_437.1", 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:55:09.165504Z" 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:55:09.444 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:55:09.445 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:09.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T22:55:09.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:09.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:09.446 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:09.499 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:09.499 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:09.530 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:09.530+0000 7fe55500f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:09.531 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:09.531+0000 7fe55500f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:09.543 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.544 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:09.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:09.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:09.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:09.548 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:09.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:09Z" 2023-09-29T22:55:09.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:09.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:09Z" } 2023-09-29T22:55:09.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:09.552 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:09.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:09.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:14.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:14.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:14.554 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:14.607 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:14.607 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:14.638 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:14.638+0000 7f9a5ecc0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:14.639 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:14.639+0000 7f9a5ecc0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:14.651 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:14Z" 2023-09-29T22:55:14.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:14.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:14Z" } 2023-09-29T22:55:14.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:14.659 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:14.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:14.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:19.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:19.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:19.661 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:19.714 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:19.715 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:19.747 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:19.747+0000 7ff742a8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:19.747 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:19.747+0000 7ff742a8b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:19.760 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:19.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:19.762 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:19.763 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:19.764 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:19.765 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:19.766 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:19.767 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:19.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:19.768 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:19Z" 2023-09-29T22:55:19.768 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:19.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:19Z" } 2023-09-29T22:55:19.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:19.769 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:19.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:19.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:24.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:24.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:24.771 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:24.823 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:24.823 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:24.855 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:24.854+0000 7f8e771af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:24.855 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:24.855+0000 7f8e771af880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:24.868 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:24.868 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:24.868 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:24.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:24.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:24.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:24.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:24.873 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:24Z" 2023-09-29T22:55:24.874 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:24.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:24Z" } 2023-09-29T22:55:24.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:24.876 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:24.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:24.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:29.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:29.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:29.877 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:29.930 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:29.930 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:29.961 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:29.961+0000 7fddb0e99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:29.962 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:29.962+0000 7fddb0e99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:29.975 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:29.975 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:29.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:29.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:29.978 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:29Z" 2023-09-29T22:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:29.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:29Z" } 2023-09-29T22:55:29.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:29.982 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:29.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:29.983 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:34.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:34.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:34.984 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:35.038 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:35.039 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:35.070 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:35.070+0000 7fa98b4c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:35.071 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:35.071+0000 7fa98b4c0880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:35.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:35.084 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:35.085 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:35.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:35.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:35.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:35.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:35.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:35.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:35.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:35.093 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:35Z" 2023-09-29T22:55:35.093 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:35.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:35Z" } 2023-09-29T22:55:35.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:35.094 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:35.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:35.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:40.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:40.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:40.096 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:40.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:40.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:40.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:40.179+0000 7fa1b6a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:40.180 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:40.180+0000 7fa1b6a19880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:40.193 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:40Z" 2023-09-29T22:55:40.200 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:40.200 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:40Z" } 2023-09-29T22:55:40.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:40.201 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:40.201 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:40.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:45.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:45.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:45.202 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:45.257 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:45.258 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:45.289 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:45.289+0000 7f9b5e00f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:45.290 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:45.290+0000 7f9b5e00f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:45.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:45.303 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:45.304 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:45.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:45.307 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:45.308 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:45.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:45.310 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:45Z" 2023-09-29T22:55:45.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:45.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:45Z" } 2023-09-29T22:55:45.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:45.311 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:45.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:45.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:50.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:50.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:50.313 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:50.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:50.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:50.399 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:50.399+0000 7fee27224880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:50.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:50.400+0000 7fee27224880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:50.413 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:50.413 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:50.413 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:50.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:50.413 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:50.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:50.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:50.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:50.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:50.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:50.420 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:50.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:50.420 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:50Z" 2023-09-29T22:55:50.420 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:50.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:50Z" } 2023-09-29T22:55:50.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:50.421 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:50.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:50.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:55:55.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:55.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:55.423 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:55.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:55.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:55.507 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:55.507+0000 7fab9534b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:55.508 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:55:55.508+0000 7fab9534b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:55:55.521 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:55:55.522 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:55.523 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:55.524 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:55:55.525 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:55.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:55:55Z" 2023-09-29T22:55:55.527 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:55:55.528 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:55Z" } 2023-09-29T22:55:55.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:55:55.529 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:55.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:55:55.529 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:00.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:00.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:00.531 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:00.584 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:00.584 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:00.615 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:00.615+0000 7f8b65f29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:00.616 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:00.616+0000 7f8b65f29880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:00.629 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:00.629 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:00.629 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:00.629 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:00.629 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:00.630 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:00.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:00.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:00Z" 2023-09-29T22:56:00.636 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:00.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:00Z" } 2023-09-29T22:56:00.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:00.637 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:00.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:00.637 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:05.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:05.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:05.640 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:05.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:05.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:05.725 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:05.724+0000 7fa98111c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:05.725 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:05.725+0000 7fa98111c880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:05.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:05.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:05.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:05.742 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:05.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:05.744 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:05Z" 2023-09-29T22:56:05.745 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:05.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:05Z" } 2023-09-29T22:56:05.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:05.746 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:05.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:05.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:10.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:10.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:10.748 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:10.800 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:10.801 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:10.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:10.832+0000 7f814af1a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:10.833 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:10.833+0000 7f814af1a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:10.845 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:10.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:10Z" 2023-09-29T22:56:10.852 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:10.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:10Z" } 2023-09-29T22:56:10.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:10.853 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:10.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:10.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:15.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:15.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:15.855 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:15.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:15.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:15.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:15.939+0000 7f3aa3921880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:15.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:15.940+0000 7f3aa3921880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:15.953 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:15.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:15.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:15.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:15Z" 2023-09-29T22:56:15.960 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:15.961 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:15Z" } 2023-09-29T22:56:15.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:15.962 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:15.962 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:15.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:20.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:20.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:20.964 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:21.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:21.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:21.049 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:21.049+0000 7fab24133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:21.050 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:21.050+0000 7fab24133880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:21.063 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:21.064 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:21.064 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:21.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:21.064 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:21.064 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:21.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:21Z" 2023-09-29T22:56:21.072 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:21.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:21Z" } 2023-09-29T22:56:21.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:21.075 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:21.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:21.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:26.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:26.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:26.077 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:26.130 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:26.130 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:26.161 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:26.161+0000 7feb0e811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:26.161 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:26.161+0000 7feb0e811880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:26Z" 2023-09-29T22:56:26.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:26.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:26Z" } 2023-09-29T22:56:26.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:26.184 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:26.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:26.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:31.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:31.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:31.186 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:31.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:31.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:31.274 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:31.274+0000 7f32e8521880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:31.275 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:31.275+0000 7f32e8521880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:31.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:31.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:31Z" 2023-09-29T22:56:31.295 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:31.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:31Z" } 2023-09-29T22:56:31.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:31.297 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:31.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:31.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:36.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:36.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:36.299 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:36.352 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:36.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:36.384 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:36.384+0000 7fe928a67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:36.385 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:36.385+0000 7fe928a67880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:36.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:36.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:36.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:36.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:36.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:36.404 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:36Z" 2023-09-29T22:56:36.405 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:36.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:36Z" } 2023-09-29T22:56:36.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:36.406 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:36.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:36.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:41.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:41.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:41.408 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:41.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:41.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:41.493 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:41.492+0000 7ffaf0736880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:41.493 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:41.493+0000 7ffaf0736880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:41.506 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696027481.827984_326.1", 2023-09-29T22:56:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:41Z" 2023-09-29T22:56:41.514 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:41.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696027481.827984_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:44:41.827984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:41Z" } 2023-09-29T22:56:41.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:41.516 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:41.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:41.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-29T22:56:46.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:46.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:46.518 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:46.572 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:46.572 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:46.603 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:46.603+0000 7f862a127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:46.604 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:46.604+0000 7f862a127880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:46.617 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:46.617 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.618 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:46.619 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:46.620 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:46.621 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.654382_385.1", 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:46.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:46.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:46.624 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:46Z" 2023-09-29T22:56:46.625 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:46.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.654382_385.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.654382Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:46Z" } 2023-09-29T22:56:46.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:46.626 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:46.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:46.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-29T22:56:51.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:51.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:51.629 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:51.683 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:51.683 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:51.714 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:51.714+0000 7f18e6729880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:51.715 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:51.715+0000 7f18e6729880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:51.728 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:51.728 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:51.728 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:51.728 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:51.728 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:51.729 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:51.730 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.823047_398.1", 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:51.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:51.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:51Z" 2023-09-29T22:56:51.734 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:51.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.823047_398.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.823047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:51Z" } 2023-09-29T22:56:51.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:51.736 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:51.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:51.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-29T22:56:56.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:56.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:56:56.738 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:56:56.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:56:56.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:56:56.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:56.828+0000 7fc3f3e32880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:56.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:56:56.829+0000 7fc3f3e32880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:56:56.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:56:56.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:56.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:56.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:56.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:56:56.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:56.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:56.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.823047_398.1", 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:56.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:56:56.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:56:56Z" 2023-09-29T22:56:56.849 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:56:56.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.823047_398.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.823047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:56:56Z" } 2023-09-29T22:56:56.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:56:56.851 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:56.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:56:56.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-29T22:57:01.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:01.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:01.853 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:01.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:01.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:01.968 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:01.968+0000 7f2719212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:01.969 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:01.969+0000 7f2719212880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:01.982 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:01.982 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:01.982 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:01.983 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:01.984 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:01.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.724469_388.1", 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-29T22:57:01.986 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:01.987 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:01.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:01.989 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:01Z" 2023-09-29T22:57:01.989 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:01.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.724469_388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.724469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:01Z" } 2023-09-29T22:57:01.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.724469_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.724469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:01.990 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:01.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.724469_388.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:01.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.724469_388.1 2023-09-29T22:57:06.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:06.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:06.992 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:07.042 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:07.042 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:07.071 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:07.071+0000 7f22d1526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:07.072 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:07.072+0000 7f22d1526880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:07.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:07.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:07.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:07.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:07.089 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:07.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:07Z" 2023-09-29T22:57:07.091 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:07.092 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:07Z" } 2023-09-29T22:57:07.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:07.092 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:07.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:07.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:57:12.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:12.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:12.094 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:12.159 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:12.159 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:12.192 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:12.192+0000 7f89e900f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:12.193 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:12.193+0000 7f89e900f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:12.205 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.823047_398.1", 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:12Z" 2023-09-29T22:57:12.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:12.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.823047_398.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.823047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:12Z" } 2023-09-29T22:57:12.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:12.213 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:12.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:12.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-29T22:57:17.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:17.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:17.214 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:17.265 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:17.265 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:17.294 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:17.294+0000 7fd9bba14880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:17.295 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:17.295+0000 7fd9bba14880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:17.307 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:17.310 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:17.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.642877_382.1", 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-29T22:57:17.312 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:17.313 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.314 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:17.315 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:17.315 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:17.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:17.315 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:17Z" 2023-09-29T22:57:17.315 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:17.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.642877_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.642877Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:17Z" } 2023-09-29T22:57:17.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:17.317 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:17.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:17.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-29T22:57:22.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:22.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:22.319 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:22.373 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:22.373 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:22.405 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:22.405+0000 7fc147815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:22.407 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:22.407+0000 7fc147815880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:22.441 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.910421_403.1", 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:22Z" 2023-09-29T22:57:22.450 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:22.451 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.910421_403.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.910421Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:22Z" } 2023-09-29T22:57:22.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.910421_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.910421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:22.451 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:22.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.910421_403.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:22.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.910421_403.1 2023-09-29T22:57:27.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:27.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:27.453 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:27.523 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:27.524 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:27.555 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:27.554+0000 7f8c50524880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:27.555 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:27.555+0000 7f8c50524880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:27.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:27.569 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:27.570 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:27.571 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:27.572 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.736877_389.1", 2023-09-29T22:57:27.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:27.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:27.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:27.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.736877Z", 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:27.573 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:27.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:27Z" 2023-09-29T22:57:27.575 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:27.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.736877_389.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.736877Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:27Z" } 2023-09-29T22:57:27.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.736877_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.736877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:27.576 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:27.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.736877_389.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:27.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.736877_389.1 2023-09-29T22:57:32.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:32.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:32.578 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:32.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:32.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:32.656 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:32.656+0000 7fc0dcc10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:32.657 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:32.657+0000 7fc0dcc10880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:32.670 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:32.670 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:32.670 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:32.670 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:32.670 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:32.671 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:32.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:32.673 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:32.674 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.839654_400.1", 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.839654Z", 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:32.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:32.676 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:32.677 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:32.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:32.678 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:32Z" 2023-09-29T22:57:32.678 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:32.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.839654_400.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.839654Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:32Z" } 2023-09-29T22:57:32.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.839654_400.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.839654Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:32.680 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:32.680 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.839654_400.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:32.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.839654_400.1 2023-09-29T22:57:37.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:37.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:37.682 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:37.736 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:37.736 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:37.769 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:37.768+0000 7ff4ffa9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:37.769 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:37.769+0000 7ff4ffa9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.654382_385.1", 2023-09-29T22:57:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:37.788 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:37.788 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:37.788 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:37.788 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:37.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:37.790 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:37.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:37.790 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:37Z" 2023-09-29T22:57:37.790 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:37.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.654382_385.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.654382Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:37Z" } 2023-09-29T22:57:37.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:37.791 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:37.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:37.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-29T22:57:42.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:42.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:42.793 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:42.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:42.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:42.874 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:42.874+0000 7fe823a0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:42.875 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:42.875+0000 7fe823a0f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:42.887 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.888 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:42.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:42.890 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.740786_390.1", 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:42.891 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:42.892 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:42.893 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:42.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:42.894 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:42Z" 2023-09-29T22:57:42.894 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:42.895 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.740786_390.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.740786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:42Z" } 2023-09-29T22:57:42.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:42.895 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:42.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:42.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-29T22:57:47.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:47.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:47.897 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:47.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:47.951 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:47.981 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:47.981+0000 7f07b0213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:47.982 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:47.982+0000 7f07b0213880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:47.995 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.654382_385.1", 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:48.002 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:48.002 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:48.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:48.002 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:47Z" 2023-09-29T22:57:48.002 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:48.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.654382_385.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.654382Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:47Z" } 2023-09-29T22:57:48.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:48.004 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:48.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:48.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-29T22:57:53.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:53.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:53.006 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:53.077 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:53.077 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:53.108 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:53.108+0000 7f04ca40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:53.109 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:53.109+0000 7f04ca40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:53.121 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:53.122 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:53.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:53.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:53.125 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:53.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:53.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:53.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:53.129 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:53.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:53.129 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:53Z" 2023-09-29T22:57:53.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:53.130 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:53Z" } 2023-09-29T22:57:53.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:53.130 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:53.130 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:53.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:57:58.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:58.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:57:58.132 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:57:58.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:57:58.182 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:57:58.211 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:58.211+0000 7f654069d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:58.211 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:57:58.211+0000 7f654069d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.823047_398.1", 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:57:58Z" 2023-09-29T22:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:57:58.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.823047_398.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.823047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:57:58Z" } 2023-09-29T22:57:58.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:57:58.232 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:58.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:57:58.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-29T22:58:03.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:03.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:03.233 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:03.287 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:03.287 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:03.334 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:03.334+0000 7f0377c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:03.338 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:03.337+0000 7f0377c15880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:03.352 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:03.354 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:03.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:03.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:03.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:03.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:03.359 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:03Z" 2023-09-29T22:58:03.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:03.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:03Z" } 2023-09-29T22:58:03.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:03.360 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:03.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:03.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:58:08.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:08.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:08.362 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:08.433 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:08.433 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:08.464 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:08.464+0000 7f108eb75880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:08.465 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:08.465+0000 7f108eb75880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:08.477 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:08.477 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:08.477 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:08.477 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:08.478 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:08.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.910421_403.1", 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:08.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:08.482 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:08.483 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:08Z" 2023-09-29T22:58:08.484 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:08.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.910421_403.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.910421Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:08Z" } 2023-09-29T22:58:08.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.910421_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.910421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:08.485 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:08.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.910421_403.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:08.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.910421_403.1 2023-09-29T22:58:13.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:13.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:13.487 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:13.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:13.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:13.572 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:13.572+0000 7f103d40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:13.573 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:13.573+0000 7f103d40f880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:13.586 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:13.586 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:13.586 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:13.586 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:13.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:13.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:13.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.740786_390.1", 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:13.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:13.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:13.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:13.593 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:13Z" 2023-09-29T22:58:13.593 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:13.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.740786_390.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.740786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:13Z" } 2023-09-29T22:58:13.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:13.594 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:13.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:13.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-29T22:58:18.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:18.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:18.595 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:18.643 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:18.644 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:18.673 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:18.673+0000 7fb52cf2a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:18.674 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:18.674+0000 7fb52cf2a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:18.686 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:18.686 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:18.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:18.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.654382_385.1", 2023-09-29T22:58:18.690 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:18.691 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:18.692 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:18Z" 2023-09-29T22:58:18.693 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:18.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.654382_385.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.654382Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:18Z" } 2023-09-29T22:58:18.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:18.695 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:18.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:18.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-29T22:58:23.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:23.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:23.696 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:23.744 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:23.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:23.773 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:23.773+0000 7fc080a26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:23.774 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:23.774+0000 7fc080a26880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:23.786 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:23.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:23.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:23.789 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.642877_382.1", 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:23.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:23.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:23.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:23Z" 2023-09-29T22:58:23.793 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:23.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.642877_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.642877Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:23Z" } 2023-09-29T22:58:23.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:23.795 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:23.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:23.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-29T22:58:28.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:28.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:28.797 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:28.851 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:28.851 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:28.885 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:28.885+0000 7f5ec380a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:28.887 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:28.887+0000 7f5ec380a880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:28.924 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:28.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:28.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:28.927 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:28.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:28.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:28.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:28Z" 2023-09-29T22:58:28.931 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:28.932 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:28Z" } 2023-09-29T22:58:28.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:28.932 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:28.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:28.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:58:33.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:33.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:33.934 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:33.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:33.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:34.051 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:34.051+0000 7efe7da99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:34.052 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:34.052+0000 7efe7da99880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:34.064 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.065 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:34.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:34.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.642877_382.1", 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:34.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:34.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:34Z" 2023-09-29T22:58:34.071 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:34.072 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.642877_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.642877Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:34Z" } 2023-09-29T22:58:34.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:34.073 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:34.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:34.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-29T22:58:39.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:39.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:39.075 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:39.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:39.128 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:39.159 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:39.158+0000 7f5b00e11880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:39.159 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:39.159+0000 7f5b00e11880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:39.172 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.642877_382.1", 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:39.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:39Z" 2023-09-29T22:58:39.179 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:39.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.642877_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.642877Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:39Z" } 2023-09-29T22:58:39.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:39.180 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:39.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:39.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-29T22:58:44.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:44.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:44.181 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:44.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:44.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:44.260 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:44.260+0000 7f302228b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:44.261 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:44.261+0000 7f302228b880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:44.274 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:44.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:44.277 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:44.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.854141_401.1", 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:44.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:44.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:44Z" 2023-09-29T22:58:44.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:44.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.854141_401.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.854141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:44Z" } 2023-09-29T22:58:44.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.854141_401.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.854141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:44.283 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:44.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.854141_401.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:44.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.854141_401.1 2023-09-29T22:58:49.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:49.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:49.284 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:49.334 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:49.334 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:49.366 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:49.366+0000 7ffb11a9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:49.366 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:49.367+0000 7ffb11a9d880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:49.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.380 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:49.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:49.382 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.740786_390.1", 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:49.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:49.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:49.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:49Z" 2023-09-29T22:58:49.386 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:49.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.740786_390.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.740786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:49Z" } 2023-09-29T22:58:49.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:49.387 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:49.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:49.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-29T22:58:54.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:54.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:54.389 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:54.449 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:54.450 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:54.480+0000 7faf64467880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:54.481 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:54.481+0000 7faf64467880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:54.493 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:54.493 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:54.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:54.495 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:54.496 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:54.497 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:54.498 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.499 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:54.500 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:54.500 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:54.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:54.500 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:54Z" 2023-09-29T22:58:54.500 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:54.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:54Z" } 2023-09-29T22:58:54.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:54.501 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:54.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:54.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:58:59.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:59.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:58:59.503 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:58:59.553 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:58:59.553 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:58:59.585 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:59.585+0000 7f98e2244880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:59.586 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:58:59.586+0000 7f98e2244880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:58:59.597 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:58:59.597 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.598 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:59.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:58:59.600 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:58:59.601 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.899474_402.1", 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:59.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:58:59.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:58:59.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:58:59.605 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:58:59Z" 2023-09-29T22:58:59.605 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:58:59.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.899474_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.899474Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:58:59Z" } 2023-09-29T22:58:59.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:58:59.606 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:59.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:58:59.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-29T22:59:04.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:04.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:04.608 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:04.662 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:04.662 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:04.698 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:04.697+0000 7f22ec328880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:04.699 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:04.699+0000 7f22ec328880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:04.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:04.713 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:04.714 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:04.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:59:04.716 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.854141_401.1", 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:04.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:04.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:04.719 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:04Z" 2023-09-29T22:59:04.720 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:04.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.854141_401.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.854141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:59:04Z" } 2023-09-29T22:59:04.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.854141_401.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.854141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:59:04.722 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:04.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.854141_401.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:04.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.854141_401.1 2023-09-29T22:59:09.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:09.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:09.724 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:09.799 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:09.799 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:09.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:09.829+0000 7fb4be2bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:09.830 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:09.830+0000 7fb4be2bc880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:09.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.843 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:59:09.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:09.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:09.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028106.724469_388.1", 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-29T22:59:09.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:59:09.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:09.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:59:09.850 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:59:09.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:09.850 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:09Z" 2023-09-29T22:59:09.850 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:09.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028106.724469_388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.724469Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:59:09Z" } 2023-09-29T22:59:09.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.724469_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.724469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:59:09.851 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:09.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.724469_388.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:09.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.724469_388.1 2023-09-29T22:59:14.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:14.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:14.853 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:14.907 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:14.907 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:14.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:14.940+0000 7f887f017880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T22:59:14.941+0000 7f887f017880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:14.954 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.955 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:59:14.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:14.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:14.958 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028109.165504_437.1", 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:14.959 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:14.960 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:14Z" 2023-09-29T22:59:14.961 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:14.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028109.165504_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:09.165504Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:59:14Z" } 2023-09-29T22:59:14.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028109.165504_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:09.165504Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:59:14.963 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:14.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:14.963 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T22:59:14.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T22:59:14.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:14.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:14.963 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:15.233 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.234 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:15.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:59:15.280+0000 7fef43415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:15.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T22:59:15.281+0000 7fef43415880 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:15.317 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T22:59:15.317 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T22:59:15.317 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T22:59:15.317 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:59:15.318 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:15.319 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:15.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028109.165504_437.1", 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:59:15.321 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T22:59:15.322 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T22:59:15.323 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T22:59:15.324 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T22:59:15.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:15.324 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T22:59:15Z" 2023-09-29T22:59:15.324 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T22:59:15.327 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028109.165504_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:09.165504Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:59:15Z" } 2023-09-29T22:59:15.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028109.165504_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:09.165504Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T22:59:15.328 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:15.328 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T22:59:15.328 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T22:59:15.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-52', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-52', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.328 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-52 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:59:15.377 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.377 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:15.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-53', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-53', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.478 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-53 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:59:15.524 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.525 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:15.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-54', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-54', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:59:15.610 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-54 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:59:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.656 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:15.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:15.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:15.753 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:15.795 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.796 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:15.862 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:59:15.862 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.29", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000078", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.history", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.23", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000070", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000104", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-29T22:59:15.863 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000026", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.8", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:15.864 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000028", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:3", 2023-09-29T22:59:15.865 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.30", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:15.866 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000040", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.6", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000015", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:15.867 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000069", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000095", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000003", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000047", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:15.868 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000035", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000037", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:15.869 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000024", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000000", 2023-09-29T22:59:15.870 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000031", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.3", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:0", 2023-09-29T22:59:15.871 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:15.872 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "data_loggenerations_metadata", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000076", 2023-09-29T22:59:15.873 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000113", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000005", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000011", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000058", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000052", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.21", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:15.874 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:15.875 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000088", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000080", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000090", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:15.876 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000110", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000096", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:15.877 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000087", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000008", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:15.878 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000006", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000029", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:15.879 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000089", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000022", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:1", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:15.880 INFO:teuthology.orchestra.run.smithi175.stdout: "bilog.trim", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000016", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000041", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:15.881 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000018", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000092", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-29T22:59:15.882 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.27", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.4", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:15.883 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000014", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000112", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000007", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000021", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:15.884 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.16", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000064", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.2.w7s8+TIelzD.0", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000071", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:15.885 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000074", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.0.QYJY6t9SUVl.0", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:15.886 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000081", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000009", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000121", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.1", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:15.887 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.17", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000125", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000082", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:15.888 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000105", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000059", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000077", 2023-09-29T22:59:15.889 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000032", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.3.4KVdP9WnyOm.0", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.24", 2023-09-29T22:59:15.890 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000053", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000091", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:15.891 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1.retry", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000065", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000083", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000010", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:15.892 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.7", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.3", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:15.893 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000045", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000002", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:15.894 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.3", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.12", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000116", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:6", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:15.895 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.25", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000034", 2023-09-29T22:59:15.896 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000101", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.13", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:15.897 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000079", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000049", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000117", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000044", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:59:15.898 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.14", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000066", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:15.899 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000068", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000085", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000073", 2023-09-29T22:59:15.900 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000038", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.10", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000118", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-29T22:59:15.901 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000036", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000103", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000119", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1.retry", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000098", 2023-09-29T22:59:15.902 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000027", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000019", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.3", 2023-09-29T22:59:15.903 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.1.V9YRRUmTpg1.0", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000039", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:15.904 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2.retry", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2.retry", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:15.905 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000100", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000093", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:15.906 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0.retry", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000004", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000063", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000122", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000057", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:15.907 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0.retry", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000054", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000114", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.2", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:2", 2023-09-29T22:59:15.908 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.0", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000012", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-29T22:59:15.909 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000084", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000043", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:4", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.11", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:15.910 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000111", 2023-09-29T22:59:15.911 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000048", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000020", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:15.912 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000099", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000056", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.19", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.28", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:15.913 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000072", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000062", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:9", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000115", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.1", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000033", 2023-09-29T22:59:15.914 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000025", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000123", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:15.915 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.18", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000108", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1", 2023-09-29T22:59:15.916 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000094", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000060", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000109", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.15", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:15.917 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000013", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:15.918 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000051", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000106", 2023-09-29T22:59:15.919 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000017", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:15.920 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000023", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:15.921 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000107", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000046", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.26", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.22", 2023-09-29T22:59:15.922 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.2", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000067", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:15.923 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.9", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000086", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.5", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:15.924 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.20", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:15.925 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000050", 2023-09-29T22:59:15.926 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000102", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:7", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000124", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:15.927 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000001", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000126", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:15.928 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:8", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000055", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:15.929 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000061", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:10", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000097", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:5", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:15.930 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000042", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000075", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000120", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000030", 2023-09-29T22:59:15.931 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:15.932 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:15.932 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9" 2023-09-29T22:59:15.932 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:59:15.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:59:15.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:15.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:15.933 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:15.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:15.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:16.038 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2023-09-29T22:59:16.038 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000078", 2023-09-29T22:59:16.038 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.history", 2023-09-29T22:59:16.038 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000070", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000104", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000026", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:16.039 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.8", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000028", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:16.040 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:16.041 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:16.042 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:16.042 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:16.042 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000040", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.6", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:16.043 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000015", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000069", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000095", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000003", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000047", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:16.044 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000035", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000037", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:16.045 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:16.046 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000024", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000000", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000031", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:16.047 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:16.048 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "data_loggenerations_metadata", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:16.049 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000076", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000113", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000005", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000011", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:16.050 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000058", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000052", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:16.051 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000088", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000080", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000090", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000110", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:16.053 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000096", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.3", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:16.054 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000087", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000008", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0.retry", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:16.055 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000006", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000029", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:16.056 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000089", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000022", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bilog.trim", 2023-09-29T22:59:16.057 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000016", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000041", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000018", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:16.058 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000092", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.4", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:16.059 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000014", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000112", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000007", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000021", 2023-09-29T22:59:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000064", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000071", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000074", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000081", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000009", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000121", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000125", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1.retry", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000082", 2023-09-29T22:59:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000105", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000059", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000077", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000032", 2023-09-29T22:59:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000053", 2023-09-29T22:59:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000091", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000065", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2", 2023-09-29T22:59:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000083", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000010", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.7", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.3", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.2", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000045", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000002", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000116", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000034", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000101", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.1", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000079", 2023-09-29T22:59:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000049", 2023-09-29T22:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000117", 2023-09-29T22:59:16.074 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000044", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000066", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:16.075 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000068", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000085", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000073", 2023-09-29T22:59:16.076 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000038", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.10", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000118", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-29T22:59:16.077 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000036", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000103", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.3", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000119", 2023-09-29T22:59:16.078 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1.retry", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000098", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.0", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:16.079 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000027", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000019", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.3", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:16.080 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000039", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2.retry", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:59:16.081 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000100", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:16.082 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000093", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0.retry", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000004", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000063", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000122", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000057", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000054", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000114", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.2", 2023-09-29T22:59:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000012", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000084", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000043", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.11", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0.85QjD62xz2M.0", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000111", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000048", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000020", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000099", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000056", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:16.089 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000072", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000062", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000115", 2023-09-29T22:59:16.090 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.1", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000033", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:16.091 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000025", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000123", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000108", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000094", 2023-09-29T22:59:16.092 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000060", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000109", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:16.093 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000013", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:16.094 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000051", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000106", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000017", 2023-09-29T22:59:16.095 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000023", 2023-09-29T22:59:16.096 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000107", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000046", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:16.097 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:16.098 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000067", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:16.099 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1.gEURCYoqqI1.0", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.9", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000086", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "sync.error-log.5", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2.3y0gFtudtMP.0", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:16.100 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:16.101 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:16.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:16.103 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000050", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000102", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2.retry", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000124", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000001", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000126", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000055", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000061", 2023-09-29T22:59:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000097", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000042", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000075", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000120", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000030", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8" 2023-09-29T22:59:16.109 INFO:teuthology.orchestra.run.smithi184.stdout:] 2023-09-29T22:59:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-09-29T22:59:16.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.110 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:16.154 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:16.154 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:16.219 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:59:16.219 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000078", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.history", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000070", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000104", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-29T22:59:16.220 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000026", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:16.221 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.8", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000028", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:16.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:16.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000040", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.6", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:16.224 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000015", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000069", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000095", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000003", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000047", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:16.225 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000035", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000037", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:16.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:2", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.2.jkVBsvUHjGr.0", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000024", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:16.227 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000000", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000031", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.3", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:16.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:4", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "data_loggenerations_metadata", 2023-09-29T22:59:16.229 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000076", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000113", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000005", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000011", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000058", 2023-09-29T22:59:16.230 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000052", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:16.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000088", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000080", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000090", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000110", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:16.232 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000096", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.3", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:16.233 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000087", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000008", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0.retry", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:16.234 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000006", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000029", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000089", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000022", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:16.235 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:10", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bilog.trim", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000016", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:16.236 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000041", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:16.237 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000018", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000092", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:16.238 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.4", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000014", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000112", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:16.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000007", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000021", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000064", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000071", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000074", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-29T22:59:16.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:16.241 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000081", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000009", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000121", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.1", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000125", 2023-09-29T22:59:16.242 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1.retry", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000082", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:7", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000105", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000059", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-29T22:59:16.243 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000077", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000032", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:16.244 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:16.245 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000053", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000091", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1.retry", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000065", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000083", 2023-09-29T22:59:16.246 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000010", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.1.i6/TT7IApej.0", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.7", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.3", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:3", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-29T22:59:16.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "mdlog.sync-status.shard.2", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-29T22:59:16.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000045", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000002", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.3", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000116", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:16.249 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:6", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000034", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-29T22:59:16.250 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:8", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000101", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "mdlog.sync-status.shard.1", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-29T22:59:16.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000079", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000049", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000117", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000044", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-29T22:59:16.252 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:1", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000066", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000068", 2023-09-29T22:59:16.253 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000085", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-29T22:59:16.254 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000073", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000038", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.10", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000118", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-29T22:59:16.255 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000036", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000103", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "mdlog.sync-status.shard.3", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-29T22:59:16.256 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000119", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000098", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-29T22:59:16.257 INFO:teuthology.orchestra.run.smithi175.stdout: "mdlog.sync-status.shard.0", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000027", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "mdlog.sync-status", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000019", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:5", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-29T22:59:16.258 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000039", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2.retry", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-29T22:59:16.259 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-29T22:59:16.260 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000100", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:16.261 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000093", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000004", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000063", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000122", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-29T22:59:16.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000057", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0.retry", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000054", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000114", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-29T22:59:16.263 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.2", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.0", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000012", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-29T22:59:16.264 INFO:teuthology.orchestra.run.smithi175.stdout: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000084", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000043", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-29T22:59:16.265 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000111", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000048", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-29T22:59:16.266 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000020", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:9", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000099", 2023-09-29T22:59:16.267 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000056", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000072", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-29T22:59:16.268 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000062", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000115", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.1", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000033", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-29T22:59:16.269 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000025", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000123", 2023-09-29T22:59:16.270 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000108", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1", 2023-09-29T22:59:16.271 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000094", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000060", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000109", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-29T22:59:16.272 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000013", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-29T22:59:16.273 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000051", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000106", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000017", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-29T22:59:16.274 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000023", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-29T22:59:16.275 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000107", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000046", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-29T22:59:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-29T22:59:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.2", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000067", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-29T22:59:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.9", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000086", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "sync.error-log.5", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-29T22:59:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-29T22:59:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-29T22:59:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000050", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000102", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2.retry", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-29T22:59:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000124", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000001", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-29T22:59:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "data_log.0.zEAIIZaW88R.0", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000126", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000055", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-29T22:59:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000061", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000097", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000042", 2023-09-29T22:59:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:0", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000075", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000120", 2023-09-29T22:59:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "obj_delete_at_hint.0000000030", 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9" 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:59:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: 2023-09-29T22:59:16.288 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2023-09-29T22:59:16.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.289 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:16.339 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:16.340 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:16.402 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T22:59:16.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000002904", 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:59:15.714290Z" 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005580", 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:59:15.715237Z" 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:16.403 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006707", 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:59:15.716232Z" 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001634", 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:59:15.718951Z" 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T22:59:16.404 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T22:59:16.405 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:16.405 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-29T22:59:16.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:16.406 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:16.455 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:16.455 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:16.497 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:16.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:16.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:16.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:16.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:16.502 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:16.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:16.504 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:16Z" 2023-09-29T22:59:16.504 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:16.505 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:16Z" } 2023-09-29T22:59:16.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:16.505 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:16.505 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:16.505 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:21.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:21.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:21.507 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:21.562 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:21.562 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:21.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:21.609 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:21.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:21.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:21.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:21Z" 2023-09-29T22:59:21.613 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:21.614 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:21Z" } 2023-09-29T22:59:21.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:21.615 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:21.615 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:21.615 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:26.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:26.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:26.616 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:26.673 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:26.673 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:26.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:26.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:26.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:26.725 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.726 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:26.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:26.727 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:26.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:26.727 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:26Z" 2023-09-29T22:59:26.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:26.728 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:26Z" } 2023-09-29T22:59:26.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:26.728 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:26.728 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:26.728 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:31.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:31.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:31.730 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:31.789 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:31.789 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:31.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:31.835 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:31.835 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:31.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:31.837 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:31.838 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:31.839 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.840 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:31.841 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:31.841 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:31.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:31.841 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:31Z" 2023-09-29T22:59:31.841 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:31.842 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:31Z" } 2023-09-29T22:59:31.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:31.842 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:31.842 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:31.842 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:36.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:36.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:36.844 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:36.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:36.903 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:36.950 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:36.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:36.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:36.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:36.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:36.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:36.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:36.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:36.956 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:36Z" 2023-09-29T22:59:36.957 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:36.958 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:36Z" } 2023-09-29T22:59:36.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:36.959 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:36.959 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:36.959 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:41.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:41.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:41.961 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:42.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:42.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:42.065 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:42.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:42.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:42.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:42.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:42.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:42.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:42.072 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:42Z" 2023-09-29T22:59:42.072 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:42.073 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:42Z" } 2023-09-29T22:59:42.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:42.073 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:42.073 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:42.073 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:47.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:47.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:47.075 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:47.131 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:47.131 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:47.180 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:47.180 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:47.180 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:47.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:47.180 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:47.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:47.182 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:47.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:47.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:47.185 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:47.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:47.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:47.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:47.187 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:47.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:47.188 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:47Z" 2023-09-29T22:59:47.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:47.189 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:47Z" } 2023-09-29T22:59:47.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:47.189 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:47.189 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:47.190 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:52.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:52.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:52.191 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:52.294 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:52.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:52.296 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:52.298 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:52.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:52.300 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:52.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:52.301 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:52Z" 2023-09-29T22:59:52.301 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:52.302 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:52Z" } 2023-09-29T22:59:52.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:52.302 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:52.302 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:52.302 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T22:59:57.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:57.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:57.304 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:57.361 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:57.361 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:57.411 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T22:59:57.411 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:57.412 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:57.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T22:59:57.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T22:59:57.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T22:59:57.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T22:59:57Z" 2023-09-29T22:59:57.417 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T22:59:57.418 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:57Z" } 2023-09-29T22:59:57.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:59:57.419 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T22:59:57.419 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T22:59:57.419 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:02.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:02.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:02.420 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:02.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:02.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:02.524 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:02.524 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:02.524 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:02.525 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:02.526 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:02.527 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:02.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:02Z" 2023-09-29T23:00:02.530 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:02.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:02Z" } 2023-09-29T23:00:02.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:02.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:02.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:02.532 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:07.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:07.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:07.533 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:07.590 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:07.591 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:07.643 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:07.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:07.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:07.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:07.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:07.648 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:07Z" 2023-09-29T23:00:07.649 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:07.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:07Z" } 2023-09-29T23:00:07.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:07.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:07.651 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:07.651 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:12.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:12.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:12.653 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:12.708 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:12.708 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:12.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:12.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:12.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:12.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:12.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:12Z" 2023-09-29T23:00:12.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:12.762 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:12Z" } 2023-09-29T23:00:12.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:12.763 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:12.763 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:12.763 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:17.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:17.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:17.764 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:17.818 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:17.819 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:17.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:17.866 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:17.866 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:17.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:17.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:17.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:17.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:17.870 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.871 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:17.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:17.872 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:17.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:17.872 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:17Z" 2023-09-29T23:00:17.872 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:17.873 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:17Z" } 2023-09-29T23:00:17.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:17.873 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:17.873 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:17.873 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:22.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:22.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:22.875 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:22.931 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:22.931 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:22.980 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:22.980 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:22.980 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:22.980 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:22.981 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:22.982 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:22.983 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:22.985 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:22.986 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:22.987 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:22.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:22.988 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:22Z" 2023-09-29T23:00:22.988 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:22.989 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:22Z" } 2023-09-29T23:00:22.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:22.989 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:22.989 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:22.989 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:27.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:27.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:27.991 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:28.045 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:28.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:28.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:28Z" 2023-09-29T23:00:28.100 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:28.101 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:28Z" } 2023-09-29T23:00:28.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:28.101 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:28.101 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:28.101 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:33.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:33.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:33.103 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:33.157 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:33.158 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:33.205 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:33.212 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:33Z" 2023-09-29T23:00:33.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:33.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:33Z" } 2023-09-29T23:00:33.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:33.213 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:33.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:33.213 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:38.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:38.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:38.214 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:38.269 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:38.270 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:38.317 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:38.317 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:38.317 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:38.318 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:38.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:38.320 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:38.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:38.322 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:38.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:38.323 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:38.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:38.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:38.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.324 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:38.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:38.325 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:38.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:38.325 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:38Z" 2023-09-29T23:00:38.325 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:38.326 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:38Z" } 2023-09-29T23:00:38.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:38.326 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:38.327 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:38.327 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:43.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:43.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:43.328 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:43.382 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:43.382 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:43.429 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:43Z" 2023-09-29T23:00:43.436 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:43.437 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:43Z" } 2023-09-29T23:00:43.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:43.438 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:43.438 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:43.438 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:48.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:48.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:48.440 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:48.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:48.495 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:48.542 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:48.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:48.544 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:48.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:48.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:48.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:48.548 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:48Z" 2023-09-29T23:00:48.548 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:48.549 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:48Z" } 2023-09-29T23:00:48.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:48.549 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:48.549 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:48.549 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:53.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:53.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:53.551 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:53.606 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:53.606 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:53.654 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:53.654 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:53.654 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:53.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:53.654 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:53.655 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:53.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:53.658 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:53.658 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:53.658 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:53.658 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:53.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:53.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:53.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:53.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:53.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:53.663 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:53Z" 2023-09-29T23:00:53.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:53.665 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:53Z" } 2023-09-29T23:00:53.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:53.665 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:53.665 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:53.665 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:00:58.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:58.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:00:58.667 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:00:58.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:58.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:58.769 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:00:58.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:00:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:00:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:00:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:00:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:00:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:00:58Z" 2023-09-29T23:00:58.776 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:00:58.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:00:58Z" } 2023-09-29T23:00:58.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:00:58.778 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:00:58.778 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:00:58.778 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:03.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:03.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:03.779 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:03.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:03.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:03.884 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:03.884 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:03.884 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:03.884 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:03.884 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:03.885 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:03.886 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:03.887 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:03.888 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:03.889 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:03Z" 2023-09-29T23:01:03.890 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:03.891 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:03Z" } 2023-09-29T23:01:03.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:03.892 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:03.892 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:03.892 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:08.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:08.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:08.893 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:08.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:08.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:08.999 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:08.999 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:08.999 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:08.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:09.000 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:09.001 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:09.002 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:09.003 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:09.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:09.005 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:09.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:09.005 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:08Z" 2023-09-29T23:01:09.005 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:09.006 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:08Z" } 2023-09-29T23:01:09.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:09.006 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:09.006 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:09.006 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:14.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:14.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:14.008 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:14.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:14.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:14Z" 2023-09-29T23:01:14.116 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:14.117 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:14Z" } 2023-09-29T23:01:14.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:14.117 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:14.117 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:14.117 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:19.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:19.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:19.118 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:19.174 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:19.174 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:19.221 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:19.221 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:19.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:19.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:19.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:19.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:19.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:19.227 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:19.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:19.228 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:19Z" 2023-09-29T23:01:19.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:19.229 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:19Z" } 2023-09-29T23:01:19.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:19.229 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:19.229 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:19.229 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:24.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:24.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:24.231 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:24.286 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:24.286 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:24.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:24.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:24Z" 2023-09-29T23:01:24.341 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:24.342 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:24Z" } 2023-09-29T23:01:24.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:24.343 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:24.343 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:24.343 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:29.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:29.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:29.345 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:29.399 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:29.400 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:29.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:29.457 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:29.457 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:29.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:29.457 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:29Z" 2023-09-29T23:01:29.457 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:29.458 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:29Z" } 2023-09-29T23:01:29.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:29.458 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:29.459 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:29.459 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:34.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:34.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:34.460 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:34.515 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:34.516 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:34.563 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:34.565 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:34.566 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:34.567 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:34.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:34.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:34.570 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:34.570 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:34.570 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:34.570 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:34.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:34.571 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:34.573 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:34.574 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:34Z" 2023-09-29T23:01:34.575 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:34.576 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:34Z" } 2023-09-29T23:01:34.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:34.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:34.577 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:34.577 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:39.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:39.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:39.581 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:39.636 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:39.636 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:39.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:39.684 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:39.684 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:39.684 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:39.684 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:39.685 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:39.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:39.687 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:39.688 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:39.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:39Z" 2023-09-29T23:01:39.690 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:39.691 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:39Z" } 2023-09-29T23:01:39.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:39.692 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:39.692 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:39.692 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:44.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:44.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:44.694 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:44.749 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:44.749 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:44.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:44.797 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.798 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:44.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:44.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:44.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:44.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:44.803 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:44.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:44.805 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:44Z" 2023-09-29T23:01:44.805 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:44.806 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:44Z" } 2023-09-29T23:01:44.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:44.806 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:44.806 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:44.807 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:49.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:49.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:49.808 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:49.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:49.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:49.911 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:49Z" 2023-09-29T23:01:49.918 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:49.919 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:49Z" } 2023-09-29T23:01:49.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:49.919 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:49.920 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:49.920 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:01:54.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:54.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:01:54.921 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:01:54.976 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:01:54.977 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:01:55.024 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:01:55.024 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:01:55.025 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:55.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:55.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:55.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:01:55Z" 2023-09-29T23:01:55.030 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:01:55.031 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:01:55Z" } 2023-09-29T23:01:55.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:01:55.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:01:55.032 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:01:55.032 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:00.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:00.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:00.033 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:00.088 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:00.089 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:00.136 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:00.136 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:00.137 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:00.138 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:00.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:00.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:00Z" 2023-09-29T23:02:00.142 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:00.143 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:00Z" } 2023-09-29T23:02:00.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:00.144 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:00.144 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:00.144 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:05.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:05.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:05.145 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:05.200 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:05.200 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.249 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:05.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:05.251 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:05.252 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:05.253 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:05.254 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:05Z" 2023-09-29T23:02:05.255 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:05.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:05Z" } 2023-09-29T23:02:05.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:05.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:05.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:05.256 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:10.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:10.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:10.257 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:10.312 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:10.313 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:10.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:10.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:10.363 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:10.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:10.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:10.366 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:10.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:10.367 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:10Z" 2023-09-29T23:02:10.367 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:10.368 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:10Z" } 2023-09-29T23:02:10.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:10.368 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:10.368 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:10.368 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:15.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:15.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:15.370 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:15.424 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:15.424 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:15.474 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:15.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:15.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:15.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:15.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:15.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:15.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:15.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:15.482 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:15.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:15.482 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:15Z" 2023-09-29T23:02:15.482 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:15.483 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:15Z" } 2023-09-29T23:02:15.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:15.483 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:15.483 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:15.483 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:20.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:20.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:20.485 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:20.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:20.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:20.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:20.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:20.589 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:20Z" 2023-09-29T23:02:20.593 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:20.594 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:20Z" } 2023-09-29T23:02:20.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:20.595 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:20.595 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:20.595 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:25.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:25.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:25.596 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:25.651 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:25.651 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:25.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:25.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:25.702 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:25.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:25.704 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:25.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:25.706 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:25Z" 2023-09-29T23:02:25.706 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:25.707 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:25Z" } 2023-09-29T23:02:25.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:25.707 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:25.707 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:25.707 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:30.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:30.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:30.709 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:30.763 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:30.764 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.812 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:30.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:30.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:30.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:30.816 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:30.817 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:30Z" 2023-09-29T23:02:30.818 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:30.819 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:30Z" } 2023-09-29T23:02:30.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:30.819 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:30.819 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:30.819 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:35.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:35.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:35.821 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:35.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:35.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:35.924 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:35.926 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.927 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:35.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:35Z" 2023-09-29T23:02:35.930 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:35.931 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:35Z" } 2023-09-29T23:02:35.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:35.931 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:35.932 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:35.932 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:40.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.933 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:40.989 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:40.989 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:41.037 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:41.037 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:41.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:41.039 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:41.041 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:41.042 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:41.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:41Z" 2023-09-29T23:02:41.044 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:41.045 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:41Z" } 2023-09-29T23:02:41.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:41.046 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:41.046 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:41.046 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:46.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:46.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:46.048 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:46.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:46.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:46.150 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:46.151 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:46.152 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:46.153 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:46.154 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:46.155 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:46.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:46.156 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:46Z" 2023-09-29T23:02:46.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:46.157 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:46Z" } 2023-09-29T23:02:46.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:46.157 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:46.157 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:46.157 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:51.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:51.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:51.160 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:51.214 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:51.214 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:51.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:51.264 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:51.265 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:51.266 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:51.267 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:51.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:51.269 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:51Z" 2023-09-29T23:02:51.269 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:51.270 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:51Z" } 2023-09-29T23:02:51.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:51.270 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:51.270 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:51.271 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:02:56.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:56.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:56.272 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:56.327 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:56.327 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.376 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:56.377 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:56.378 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:56.379 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:02:56.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.381 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:02:56.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:02:56.382 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:02:56.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:02:56.382 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:02:56Z" 2023-09-29T23:02:56.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:02:56.383 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:56Z" } 2023-09-29T23:02:56.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:02:56.383 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:02:56.383 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:02:56.384 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:01.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:01.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:01.385 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:01.439 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:01.439 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:01.487 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:01.487 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:01.488 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:01.489 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:01.490 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:01.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:01.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:01Z" 2023-09-29T23:03:01.493 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:01.494 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:01Z" } 2023-09-29T23:03:01.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:01.495 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:01.495 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:01.495 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:06.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:06.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:06.497 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:06.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:06.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:06.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:06.601 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:06.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:06.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:06.604 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:06.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:06.606 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:06.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:06.606 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:06Z" 2023-09-29T23:03:06.606 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:06.607 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:06Z" } 2023-09-29T23:03:06.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:06.607 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:06.607 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:06.607 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:11.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:11.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:11.609 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:11.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:11.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:11.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:11Z" 2023-09-29T23:03:11.720 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:11.721 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:11Z" } 2023-09-29T23:03:11.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:11.721 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:11.721 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:11.721 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:16.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:16.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:16.723 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:16.778 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:16.778 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:16.827 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:16.827 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:16.827 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:16.827 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:16.827 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:16.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:16.829 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:16.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:16Z" 2023-09-29T23:03:16.834 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:16.835 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:16Z" } 2023-09-29T23:03:16.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:16.836 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:16.836 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:16.836 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:21.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:21.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:21.837 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:21.892 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:21.892 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:21.940 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:21.940 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:21.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:21.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:21.943 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:21.945 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:21Z" 2023-09-29T23:03:21.946 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:21.947 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:21Z" } 2023-09-29T23:03:21.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:21.948 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:21.948 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:21.948 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:26.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:26.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:26.949 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:27.005 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:27.005 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:27.055 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:27.055 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:27.055 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:27.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:27.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:27.057 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:27.058 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:27.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:27.060 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:27Z" 2023-09-29T23:03:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:27.062 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:27Z" } 2023-09-29T23:03:27.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:27.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:27.063 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:27.063 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:32.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:32.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:32.064 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:32.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:32.119 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:32Z" 2023-09-29T23:03:32.174 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:32.175 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:32Z" } 2023-09-29T23:03:32.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:32.175 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:32.175 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:32.175 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:37.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:37.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:37.178 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:37.232 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:37.232 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:37.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:37.281 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:37.281 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:37.281 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:37.281 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:37.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:37.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:37.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:37.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:37.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:37.290 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:37Z" 2023-09-29T23:03:37.291 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:37.292 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:37Z" } 2023-09-29T23:03:37.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:37.293 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:37.293 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:37.293 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:42.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:42.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:42.295 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:42.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:42.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:42.397 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:42.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:42.403 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:42.404 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:42.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:42.404 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:42Z" 2023-09-29T23:03:42.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:42.405 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:42Z" } 2023-09-29T23:03:42.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:42.405 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:42.405 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:42.405 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:47.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:47.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:47.407 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:47.463 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:47.463 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:47.513 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:47.514 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:47.514 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:47.514 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:47.514 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:47.514 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:47.515 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:47.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:47.517 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:47.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:47.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:47.520 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:47.521 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:47.521 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:47.521 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:47.521 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:47.521 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:47.522 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:47.522 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:47.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:47.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:47.522 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:47.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.523 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:47.523 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:47.523 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:47.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:47.524 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:47Z" 2023-09-29T23:03:47.524 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:47.525 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:47Z" } 2023-09-29T23:03:47.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:47.526 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:47.526 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:47.526 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:52.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:52.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:52.527 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:52.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:52.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:52.630 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:52.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:52.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:52.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:52.635 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:52.635 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:52.635 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.636 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:52.637 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:52Z" 2023-09-29T23:03:52.637 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:52.638 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:52Z" } 2023-09-29T23:03:52.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:52.638 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:52.638 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:52.639 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:03:57.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:57.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:03:57.640 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:03:57.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:03:57.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:03:57.742 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:03:57.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:57.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:57.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:03:57.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:57.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:03:57.748 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:03:57.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:03:57.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:57.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:03:57.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:03:57.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.751 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:03:57.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:03:57.751 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:03:57.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:03:57.752 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:03:57Z" 2023-09-29T23:03:57.752 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:03:57.753 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:03:57Z" } 2023-09-29T23:03:57.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:03:57.754 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:03:57.754 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:03:57.754 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:04:02.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:02.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:02.755 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:02.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:02.810 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:02.858 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:02.858 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:02.858 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:02.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:02.859 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:02.859 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:02.860 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:02.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:02.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:02Z" 2023-09-29T23:04:02.865 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:02.866 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:04:02Z" } 2023-09-29T23:04:02.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:04:02.867 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:04:02.867 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:04:02.867 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:04:07.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:07.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:07.869 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:07.924 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:07.925 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:07.973 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:04:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:04:07.979 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:04:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:07.980 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:07Z" 2023-09-29T23:04:07.980 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:07.981 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:04:07Z" } 2023-09-29T23:04:07.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:04:07.981 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:04:07.982 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:04:07.982 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:04:12.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:12.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:12.984 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:13.039 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:13.039 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:13.088 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:13.088 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:13.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:13.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:13.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:04:13.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:13Z" 2023-09-29T23:04:13.094 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:13.095 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:04:13Z" } 2023-09-29T23:04:13.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:04:13.096 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:04:13.096 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:04:13.096 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:04:18.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:18.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:18.098 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:18.152 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:18.152 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 11609692693196001247 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.201 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-29T23:04:18.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004319", 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:18.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005306", 2023-09-29T23:04:18.204 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-29T23:04:18.205 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:18.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2023-09-29T23:04:18.207 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2023-09-29T23:04:18.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:18.207 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:18Z" 2023-09-29T23:04:18.207 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:18.208 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11609692693196001247 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:16:54.204956Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:38:40.088667Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:20:34.216869Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:04:18Z" } 2023-09-29T23:04:18.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T23:04:18.208 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-29T23:04:18.208 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-29T23:04:18.208 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-29T23:04:23.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2023-09-29T23:04:23.216 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-55 2023-09-29T23:04:23.216 DEBUG:boto:path=/dmuhsu-55/ 2023-09-29T23:04:23.216 DEBUG:boto:auth_path=/dmuhsu-55/ 2023-09-29T23:04:23.217 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.217 DEBUG:boto:Path: /dmuhsu-55/ 2023-09-29T23:04:23.217 DEBUG:boto:Data: 2023-09-29T23:04:23.217 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.217 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.217 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.217 DEBUG:boto:Params: {} 2023-09-29T23:04:23.217 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:04:23.218 DEBUG:boto:Token: None 2023-09-29T23:04:23.218 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/ 2023-09-29T23:04:23.219 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oWpMQjz3WjmBDR6QGff4m7UkLck= 2023-09-29T23:04:23.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oWpMQjz3WjmBDR6QGff4m7UkLck=', 'Content-Length': '0'} 2023-09-29T23:04:23.246 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7bf093cdef5ac4c-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.246 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-56 2023-09-29T23:04:23.247 DEBUG:boto:path=/dmuhsu-56/ 2023-09-29T23:04:23.247 DEBUG:boto:auth_path=/dmuhsu-56/ 2023-09-29T23:04:23.247 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.247 DEBUG:boto:Path: /dmuhsu-56/ 2023-09-29T23:04:23.247 DEBUG:boto:Data: 2023-09-29T23:04:23.247 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.247 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.247 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.247 DEBUG:boto:Params: {} 2023-09-29T23:04:23.248 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:04:23.248 DEBUG:boto:Token: None 2023-09-29T23:04:23.248 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/ 2023-09-29T23:04:23.248 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vRa6hmAPEtGu5SaFzj1v2dPeQMM= 2023-09-29T23:04:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vRa6hmAPEtGu5SaFzj1v2dPeQMM=', 'Content-Length': '0'} 2023-09-29T23:04:23.298 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2a0d56bcfb76e2b-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.298 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-57 2023-09-29T23:04:23.300 DEBUG:boto:path=/dmuhsu-57/ 2023-09-29T23:04:23.300 DEBUG:boto:auth_path=/dmuhsu-57/ 2023-09-29T23:04:23.300 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.300 DEBUG:boto:Path: /dmuhsu-57/ 2023-09-29T23:04:23.300 DEBUG:boto:Data: 2023-09-29T23:04:23.300 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.300 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.300 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.301 DEBUG:boto:Params: {} 2023-09-29T23:04:23.301 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:04:23.301 DEBUG:boto:Token: None 2023-09-29T23:04:23.301 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/ 2023-09-29T23:04:23.301 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:l3ijMzIi/WMWtmJdnh7rEbHEPZM= 2023-09-29T23:04:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l3ijMzIi/WMWtmJdnh7rEbHEPZM=', 'Content-Length': '0'} 2023-09-29T23:04:23.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca4e7c5424524c48-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.337 DEBUG:boto:path=/dmuhsu-55/ 2023-09-29T23:04:23.337 DEBUG:boto:auth_path=/dmuhsu-55/ 2023-09-29T23:04:23.337 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.337 DEBUG:boto:Path: /dmuhsu-55/ 2023-09-29T23:04:23.337 DEBUG:boto:Data: 2023-09-29T23:04:23.338 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.338 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.338 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.338 DEBUG:boto:Params: {} 2023-09-29T23:04:23.338 DEBUG:boto:Token: None 2023-09-29T23:04:23.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/ 2023-09-29T23:04:23.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-29T23:04:23.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-29T23:04:23.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f7d6bd2886fef02-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.342 DEBUG:boto:path=/dmuhsu-55/obj1 2023-09-29T23:04:23.342 DEBUG:boto:auth_path=/dmuhsu-55/obj1 2023-09-29T23:04:23.342 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.342 DEBUG:boto:Path: /dmuhsu-55/obj1 2023-09-29T23:04:23.342 DEBUG:boto:Data: 2023-09-29T23:04:23.342 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.343 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.343 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.343 DEBUG:boto:Params: {} 2023-09-29T23:04:23.343 DEBUG:boto:Token: None 2023-09-29T23:04:23.343 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/obj1 2023-09-29T23:04:23.343 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eIWWFkB/lvUOyp3IWm0YD+w+/Es= 2023-09-29T23:04:23.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eIWWFkB/lvUOyp3IWm0YD+w+/Es='} 2023-09-29T23:04:23.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e2e039585245362-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.353 DEBUG:boto:path=/dmuhsu-55/ 2023-09-29T23:04:23.353 DEBUG:boto:auth_path=/dmuhsu-55/ 2023-09-29T23:04:23.353 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.353 DEBUG:boto:Path: /dmuhsu-55/ 2023-09-29T23:04:23.353 DEBUG:boto:Data: 2023-09-29T23:04:23.353 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.353 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.354 DEBUG:boto:Params: {} 2023-09-29T23:04:23.354 DEBUG:boto:Token: None 2023-09-29T23:04:23.354 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/ 2023-09-29T23:04:23.354 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-29T23:04:23.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-29T23:04:23.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089668def3248ac90-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.357 DEBUG:boto:path=/dmuhsu-55/obj2 2023-09-29T23:04:23.357 DEBUG:boto:auth_path=/dmuhsu-55/obj2 2023-09-29T23:04:23.357 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.358 DEBUG:boto:Path: /dmuhsu-55/obj2 2023-09-29T23:04:23.358 DEBUG:boto:Data: 2023-09-29T23:04:23.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.358 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.358 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.358 DEBUG:boto:Params: {} 2023-09-29T23:04:23.358 DEBUG:boto:Token: None 2023-09-29T23:04:23.358 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/obj2 2023-09-29T23:04:23.358 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3WKykTwxNpN8ntJoltAAenJGpQA= 2023-09-29T23:04:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3WKykTwxNpN8ntJoltAAenJGpQA='} 2023-09-29T23:04:23.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b70b4a2db03bc43e-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.367 DEBUG:boto:path=/dmuhsu-55/ 2023-09-29T23:04:23.367 DEBUG:boto:auth_path=/dmuhsu-55/ 2023-09-29T23:04:23.368 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.368 DEBUG:boto:Path: /dmuhsu-55/ 2023-09-29T23:04:23.368 DEBUG:boto:Data: 2023-09-29T23:04:23.368 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.368 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.368 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.368 DEBUG:boto:Params: {} 2023-09-29T23:04:23.368 DEBUG:boto:Token: None 2023-09-29T23:04:23.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/ 2023-09-29T23:04:23.369 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-29T23:04:23.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-29T23:04:23.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1d3a07bf24f3f8f-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.371 DEBUG:boto:path=/dmuhsu-55/obj3 2023-09-29T23:04:23.371 DEBUG:boto:auth_path=/dmuhsu-55/obj3 2023-09-29T23:04:23.372 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.372 DEBUG:boto:Path: /dmuhsu-55/obj3 2023-09-29T23:04:23.372 DEBUG:boto:Data: 2023-09-29T23:04:23.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.372 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.372 DEBUG:boto:Params: {} 2023-09-29T23:04:23.372 DEBUG:boto:Token: None 2023-09-29T23:04:23.373 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/obj3 2023-09-29T23:04:23.373 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PxqPUPrShGQSHab4UK4QJsxHEwY= 2023-09-29T23:04:23.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PxqPUPrShGQSHab4UK4QJsxHEwY='} 2023-09-29T23:04:23.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008965d169d199dd78-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.415 DEBUG:boto:path=/dmuhsu-55/ 2023-09-29T23:04:23.415 DEBUG:boto:auth_path=/dmuhsu-55/ 2023-09-29T23:04:23.415 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.415 DEBUG:boto:Path: /dmuhsu-55/ 2023-09-29T23:04:23.415 DEBUG:boto:Data: 2023-09-29T23:04:23.415 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.415 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.416 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.416 DEBUG:boto:Params: {} 2023-09-29T23:04:23.416 DEBUG:boto:Token: None 2023-09-29T23:04:23.416 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/ 2023-09-29T23:04:23.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-29T23:04:23.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-29T23:04:23.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fac7cf36bce219e-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.419 DEBUG:boto:path=/dmuhsu-55/obj4 2023-09-29T23:04:23.419 DEBUG:boto:auth_path=/dmuhsu-55/obj4 2023-09-29T23:04:23.419 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.419 DEBUG:boto:Path: /dmuhsu-55/obj4 2023-09-29T23:04:23.420 DEBUG:boto:Data: 2023-09-29T23:04:23.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.420 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:04:23.420 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.420 DEBUG:boto:Params: {} 2023-09-29T23:04:23.420 DEBUG:boto:Token: None 2023-09-29T23:04:23.420 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-55/obj4 2023-09-29T23:04:23.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:epm8eyr3aibbvrnPIysEhH2Er/s= 2023-09-29T23:04:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:epm8eyr3aibbvrnPIysEhH2Er/s='} 2023-09-29T23:04:23.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d52aa9ce76d067-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.432 DEBUG:boto:path=/dmuhsu-56/ 2023-09-29T23:04:23.432 DEBUG:boto:auth_path=/dmuhsu-56/ 2023-09-29T23:04:23.432 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.432 DEBUG:boto:Path: /dmuhsu-56/ 2023-09-29T23:04:23.433 DEBUG:boto:Data: 2023-09-29T23:04:23.433 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.433 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.433 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.433 DEBUG:boto:Params: {} 2023-09-29T23:04:23.433 DEBUG:boto:Token: None 2023-09-29T23:04:23.433 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/ 2023-09-29T23:04:23.433 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-29T23:04:23.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-29T23:04:23.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027fd35f6a1cf6911-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.436 DEBUG:boto:path=/dmuhsu-56/obj1 2023-09-29T23:04:23.436 DEBUG:boto:auth_path=/dmuhsu-56/obj1 2023-09-29T23:04:23.436 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.436 DEBUG:boto:Path: /dmuhsu-56/obj1 2023-09-29T23:04:23.436 DEBUG:boto:Data: 2023-09-29T23:04:23.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.437 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.437 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.437 DEBUG:boto:Params: {} 2023-09-29T23:04:23.437 DEBUG:boto:Token: None 2023-09-29T23:04:23.437 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/obj1 2023-09-29T23:04:23.437 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MjSuwqCv12JxOv2qjpSNtgmfn4Q= 2023-09-29T23:04:23.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MjSuwqCv12JxOv2qjpSNtgmfn4Q='} 2023-09-29T23:04:23.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001258b31831496167-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.445 DEBUG:boto:path=/dmuhsu-56/ 2023-09-29T23:04:23.445 DEBUG:boto:auth_path=/dmuhsu-56/ 2023-09-29T23:04:23.445 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.445 DEBUG:boto:Path: /dmuhsu-56/ 2023-09-29T23:04:23.445 DEBUG:boto:Data: 2023-09-29T23:04:23.445 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.445 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.446 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.446 DEBUG:boto:Params: {} 2023-09-29T23:04:23.446 DEBUG:boto:Token: None 2023-09-29T23:04:23.446 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/ 2023-09-29T23:04:23.446 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-29T23:04:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-29T23:04:23.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017fd30f3ff3d5cc9-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.449 DEBUG:boto:path=/dmuhsu-56/obj2 2023-09-29T23:04:23.449 DEBUG:boto:auth_path=/dmuhsu-56/obj2 2023-09-29T23:04:23.449 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.449 DEBUG:boto:Path: /dmuhsu-56/obj2 2023-09-29T23:04:23.449 DEBUG:boto:Data: 2023-09-29T23:04:23.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.449 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.449 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.449 DEBUG:boto:Params: {} 2023-09-29T23:04:23.450 DEBUG:boto:Token: None 2023-09-29T23:04:23.450 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/obj2 2023-09-29T23:04:23.450 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nqvrOzIxkKuxYjYN7GIbdy8X+kc= 2023-09-29T23:04:23.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nqvrOzIxkKuxYjYN7GIbdy8X+kc='} 2023-09-29T23:04:23.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e428b19b46a963f8-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.456 DEBUG:boto:path=/dmuhsu-56/ 2023-09-29T23:04:23.456 DEBUG:boto:auth_path=/dmuhsu-56/ 2023-09-29T23:04:23.456 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.456 DEBUG:boto:Path: /dmuhsu-56/ 2023-09-29T23:04:23.456 DEBUG:boto:Data: 2023-09-29T23:04:23.456 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.456 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.456 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.457 DEBUG:boto:Params: {} 2023-09-29T23:04:23.457 DEBUG:boto:Token: None 2023-09-29T23:04:23.457 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/ 2023-09-29T23:04:23.457 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-29T23:04:23.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-29T23:04:23.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc8917c1c6cc44b8-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.459 DEBUG:boto:path=/dmuhsu-56/obj3 2023-09-29T23:04:23.459 DEBUG:boto:auth_path=/dmuhsu-56/obj3 2023-09-29T23:04:23.459 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.459 DEBUG:boto:Path: /dmuhsu-56/obj3 2023-09-29T23:04:23.460 DEBUG:boto:Data: 2023-09-29T23:04:23.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.460 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.460 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.460 DEBUG:boto:Params: {} 2023-09-29T23:04:23.460 DEBUG:boto:Token: None 2023-09-29T23:04:23.460 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/obj3 2023-09-29T23:04:23.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1P/+rlG7FRRBclfDV526hczXhbk= 2023-09-29T23:04:23.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1P/+rlG7FRRBclfDV526hczXhbk='} 2023-09-29T23:04:23.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000075c1931c183e601-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.503 DEBUG:boto:path=/dmuhsu-56/ 2023-09-29T23:04:23.503 DEBUG:boto:auth_path=/dmuhsu-56/ 2023-09-29T23:04:23.503 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.503 DEBUG:boto:Path: /dmuhsu-56/ 2023-09-29T23:04:23.503 DEBUG:boto:Data: 2023-09-29T23:04:23.503 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.503 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.503 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.504 DEBUG:boto:Params: {} 2023-09-29T23:04:23.504 DEBUG:boto:Token: None 2023-09-29T23:04:23.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/ 2023-09-29T23:04:23.504 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-29T23:04:23.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-29T23:04:23.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001379f075074cfeb-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.507 DEBUG:boto:path=/dmuhsu-56/obj4 2023-09-29T23:04:23.507 DEBUG:boto:auth_path=/dmuhsu-56/obj4 2023-09-29T23:04:23.507 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.507 DEBUG:boto:Path: /dmuhsu-56/obj4 2023-09-29T23:04:23.507 DEBUG:boto:Data: 2023-09-29T23:04:23.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.507 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:04:23.507 DEBUG:boto:Port: 8000 2023-09-29T23:04:23.507 DEBUG:boto:Params: {} 2023-09-29T23:04:23.508 DEBUG:boto:Token: None 2023-09-29T23:04:23.508 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-56/obj4 2023-09-29T23:04:23.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Guc5d00QDtJDWLvtau2xWtuQP9M= 2023-09-29T23:04:23.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Guc5d00QDtJDWLvtau2xWtuQP9M='} 2023-09-29T23:04:23.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff1ce93489b56023-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.514 DEBUG:boto:path=/dmuhsu-57/ 2023-09-29T23:04:23.514 DEBUG:boto:auth_path=/dmuhsu-57/ 2023-09-29T23:04:23.515 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.515 DEBUG:boto:Path: /dmuhsu-57/ 2023-09-29T23:04:23.515 DEBUG:boto:Data: 2023-09-29T23:04:23.515 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.515 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.515 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.515 DEBUG:boto:Params: {} 2023-09-29T23:04:23.515 DEBUG:boto:Token: None 2023-09-29T23:04:23.515 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/ 2023-09-29T23:04:23.516 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-29T23:04:23.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-29T23:04:23.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000400facd989fc8bad-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.518 DEBUG:boto:path=/dmuhsu-57/obj1 2023-09-29T23:04:23.521 DEBUG:boto:auth_path=/dmuhsu-57/obj1 2023-09-29T23:04:23.521 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.521 DEBUG:boto:Path: /dmuhsu-57/obj1 2023-09-29T23:04:23.521 DEBUG:boto:Data: 2023-09-29T23:04:23.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.521 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.522 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.522 DEBUG:boto:Params: {} 2023-09-29T23:04:23.522 DEBUG:boto:Token: None 2023-09-29T23:04:23.522 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/obj1 2023-09-29T23:04:23.522 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1bk2icKiGo7klPNnzKQKT4aXm3I= 2023-09-29T23:04:23.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1bk2icKiGo7klPNnzKQKT4aXm3I='} 2023-09-29T23:04:23.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d19c18f5f7e8809a-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.529 DEBUG:boto:path=/dmuhsu-57/ 2023-09-29T23:04:23.530 DEBUG:boto:auth_path=/dmuhsu-57/ 2023-09-29T23:04:23.530 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.530 DEBUG:boto:Path: /dmuhsu-57/ 2023-09-29T23:04:23.530 DEBUG:boto:Data: 2023-09-29T23:04:23.530 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.530 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.530 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.530 DEBUG:boto:Params: {} 2023-09-29T23:04:23.530 DEBUG:boto:Token: None 2023-09-29T23:04:23.531 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/ 2023-09-29T23:04:23.531 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-29T23:04:23.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-29T23:04:23.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000463b97d8cbeefcb7-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.536 DEBUG:boto:path=/dmuhsu-57/obj2 2023-09-29T23:04:23.536 DEBUG:boto:auth_path=/dmuhsu-57/obj2 2023-09-29T23:04:23.536 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.536 DEBUG:boto:Path: /dmuhsu-57/obj2 2023-09-29T23:04:23.536 DEBUG:boto:Data: 2023-09-29T23:04:23.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.537 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.537 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.537 DEBUG:boto:Params: {} 2023-09-29T23:04:23.537 DEBUG:boto:Token: None 2023-09-29T23:04:23.538 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/obj2 2023-09-29T23:04:23.538 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CdWQx28KODYEdD7PGzeQdEmgEz8= 2023-09-29T23:04:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CdWQx28KODYEdD7PGzeQdEmgEz8='} 2023-09-29T23:04:23.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010b15d5a160a5cc9-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.554 DEBUG:boto:path=/dmuhsu-57/ 2023-09-29T23:04:23.554 DEBUG:boto:auth_path=/dmuhsu-57/ 2023-09-29T23:04:23.555 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.555 DEBUG:boto:Path: /dmuhsu-57/ 2023-09-29T23:04:23.555 DEBUG:boto:Data: 2023-09-29T23:04:23.555 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.555 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.555 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.555 DEBUG:boto:Params: {} 2023-09-29T23:04:23.556 DEBUG:boto:Token: None 2023-09-29T23:04:23.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/ 2023-09-29T23:04:23.556 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-29T23:04:23.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-29T23:04:23.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7b25b4afe492201-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.558 DEBUG:boto:path=/dmuhsu-57/obj3 2023-09-29T23:04:23.558 DEBUG:boto:auth_path=/dmuhsu-57/obj3 2023-09-29T23:04:23.558 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.559 DEBUG:boto:Path: /dmuhsu-57/obj3 2023-09-29T23:04:23.559 DEBUG:boto:Data: 2023-09-29T23:04:23.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.559 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.559 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.559 DEBUG:boto:Params: {} 2023-09-29T23:04:23.559 DEBUG:boto:Token: None 2023-09-29T23:04:23.559 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/obj3 2023-09-29T23:04:23.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WeIMA3Im49/IEdE6GROw8Z548qI= 2023-09-29T23:04:23.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WeIMA3Im49/IEdE6GROw8Z548qI='} 2023-09-29T23:04:23.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b1dd4fe7aa0274e-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.567 DEBUG:boto:path=/dmuhsu-57/ 2023-09-29T23:04:23.567 DEBUG:boto:auth_path=/dmuhsu-57/ 2023-09-29T23:04:23.568 DEBUG:boto:Method: HEAD 2023-09-29T23:04:23.568 DEBUG:boto:Path: /dmuhsu-57/ 2023-09-29T23:04:23.568 DEBUG:boto:Data: 2023-09-29T23:04:23.568 DEBUG:boto:Headers: {} 2023-09-29T23:04:23.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.568 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.568 DEBUG:boto:Params: {} 2023-09-29T23:04:23.569 DEBUG:boto:Token: None 2023-09-29T23:04:23.569 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/ 2023-09-29T23:04:23.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-29T23:04:23.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-29T23:04:23.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b1e922a3bd16bb-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.571 DEBUG:boto:path=/dmuhsu-57/obj4 2023-09-29T23:04:23.571 DEBUG:boto:auth_path=/dmuhsu-57/obj4 2023-09-29T23:04:23.571 DEBUG:boto:Method: PUT 2023-09-29T23:04:23.572 DEBUG:boto:Path: /dmuhsu-57/obj4 2023-09-29T23:04:23.572 DEBUG:boto:Data: 2023-09-29T23:04:23.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:04:23.572 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:04:23.572 DEBUG:boto:Port: 8001 2023-09-29T23:04:23.572 DEBUG:boto:Params: {} 2023-09-29T23:04:23.572 DEBUG:boto:Token: None 2023-09-29T23:04:23.573 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:04:23 GMT /dmuhsu-57/obj4 2023-09-29T23:04:23.573 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4bcDD8WWQJ1aEtKJUHOQSsrdHMI= 2023-09-29T23:04:23.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4bcDD8WWQJ1aEtKJUHOQSsrdHMI='} 2023-09-29T23:04:23.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f321d0151544b098-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:04:23.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:23.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:23.615 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:23.709 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:23.709 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:23.740 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:04:23.753 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:04:23.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:23.753 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:04:23.753 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:04:23.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028663.327363_455.1", 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:04:23.327363Z" 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:23.754 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:23.755 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:04:23.755 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:04:23.755 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:23.755 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:04:23.756 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:23.756 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:04:23.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:23.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:23.756 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:23.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:23.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:23.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:04:23.841+0000 7fb3ba721880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:23.842 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:04:23.842+0000 7fb3ba721880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:23.856 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:23.856 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:23.856 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:23.857 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:23.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:04:23.862 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028663.320433_452.1", 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:23.863 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:04:23.320433Z", 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:23.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.865 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:23.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:04:23.866 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:04:23.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:23.866 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:23Z" 2023-09-29T23:04:23.866 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:23.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028663.320433_452.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:04:23.320433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:04:23Z" } 2023-09-29T23:04:23.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.320433_452.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.320433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:04:23.868 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:23.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.320433_452.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:23.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028663.327363_455.1 target=1_1696028663.320433_452.1 2023-09-29T23:04:28.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:28.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:28.870 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:28.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:28.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:28.964 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:04:28.963+0000 7f0b67617880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:28.965 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:04:28.964+0000 7f0b67617880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:28.980 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:04:28.980 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:04:28.980 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:04:28.981 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:28.983 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028663.327363_455.1", 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:28.984 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:04:28.985 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:04:28.986 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:04:28Z" 2023-09-29T23:04:28.987 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:04:28.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028663.327363_455.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:04:23.327363Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:04:28Z" } 2023-09-29T23:04:28.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.327363_455.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.327363Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:04:28.989 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:28.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:28.989 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:04:28.989 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:04:28.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:28.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:04:28.990 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:04:29.042 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:29.042 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:29.106 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:29.105+0000 7fca60d26880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:29.107 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:29.106+0000 7fca60d26880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:04:29.125 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:04:29.125 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:29.126 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.127 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:04:29.128 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028663.327363_455.1", 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-29T23:04:29.129 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:04:29.130 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:04:29.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:04:29.132 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:04:29Z" 2023-09-29T23:04:29.132 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:04:29.132 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028663.327363_455.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:04:23.327363Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:04:29Z" } 2023-09-29T23:04:29.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.327363_455.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.327363Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:04:29.133 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:29.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:04:29.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:04:29.133 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-55 2023-09-29T23:04:29.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.134 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:04:29.187 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:29.187 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:29.218 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-55 2023-09-29T23:04:29.232 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-55 2023-09-29T23:04:29.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.233 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:04:29.286 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:29.287 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:29.348 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:29.347+0000 7f108a60f880 1 bucket sync caught up with source: 2023-09-29T23:04:29.348 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [00000000001.16.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.10.6, , , ] 2023-09-29T23:04:29.348 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [00000000001.16.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.10.6, , , ] 2023-09-29T23:04:29.349 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:29.347+0000 7f108a60f880 0 bucket checkpoint complete 2023-09-29T23:04:29.362 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-55 2023-09-29T23:04:29.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:04:29.363 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:04:29.415 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:04:29.415 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:04:29.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:29.448+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:34.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:34.448+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:39.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:39.449+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:44.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:44.450+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:49.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:49.450+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:54.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:54.451+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:04:59.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:04:59.451+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:04.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:04.452+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:09.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:09.452+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:14.453+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:19.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:19.454+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:24.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:24.453+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:29.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:29.454+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:34.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:34.455+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:39.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:39.456+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:44.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:44.457+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:49.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:49.457+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:54.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:54.458+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:05:59.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:05:59.458+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:04.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:04.459+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:09.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:09.459+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:14.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:14.460+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:19.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:19.461+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:24.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:24.460+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:29.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:29.461+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:34.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:34.462+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:39.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:39.463+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:44.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:44.464+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:49.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:49.464+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:54.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:54.465+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:06:59.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:06:59.465+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:04.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:04.466+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:09.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:09.466+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:14.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:14.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:19.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:19.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:24.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:24.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:29.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:29.468+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:34.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:34.468+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:39.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:39.469+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:44.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:44.470+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:49.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:49.470+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:54.471+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:07:59.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:07:59.472+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:04.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:04.472+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:09.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:09.473+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:14.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:14.474+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:19.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:19.473+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:24.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:24.474+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:29.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:29.475+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:34.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:34.475+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:39.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:39.476+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:44.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:44.477+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:49.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:49.478+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:54.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:54.478+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:08:59.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:08:59.479+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:09:04.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:04.479+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:09:09.480 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:09.480+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:09:14.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:14.481+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:09:19.481 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:19.480+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-29T23:09:24.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:24.481+0000 7ff642a44880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:09:24.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:24.481+0000 7ff642a44880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:09:24.482 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:24.481+0000 7ff642a44880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:09:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:09:24.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2023-09-29T23:09:24.504 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-58 2023-09-29T23:09:24.504 DEBUG:boto:path=/dmuhsu-58/ 2023-09-29T23:09:24.504 DEBUG:boto:auth_path=/dmuhsu-58/ 2023-09-29T23:09:24.504 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.504 DEBUG:boto:Path: /dmuhsu-58/ 2023-09-29T23:09:24.504 DEBUG:boto:Data: 2023-09-29T23:09:24.505 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.505 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.505 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.505 DEBUG:boto:Params: {} 2023-09-29T23:09:24.505 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:09:24.505 DEBUG:boto:Token: None 2023-09-29T23:09:24.505 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/ 2023-09-29T23:09:24.506 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0KTj9qj1iAGxY2pWXbOt27tAIWk= 2023-09-29T23:09:24.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0KTj9qj1iAGxY2pWXbOt27tAIWk=', 'Content-Length': '0'} 2023-09-29T23:09:24.538 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa249fc20c5a46cb-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.539 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-59 2023-09-29T23:09:24.539 DEBUG:boto:path=/dmuhsu-59/ 2023-09-29T23:09:24.539 DEBUG:boto:auth_path=/dmuhsu-59/ 2023-09-29T23:09:24.539 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.539 DEBUG:boto:Path: /dmuhsu-59/ 2023-09-29T23:09:24.540 DEBUG:boto:Data: 2023-09-29T23:09:24.540 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.540 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.540 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.540 DEBUG:boto:Params: {} 2023-09-29T23:09:24.540 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:09:24.540 DEBUG:boto:Token: None 2023-09-29T23:09:24.541 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/ 2023-09-29T23:09:24.541 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x+UcKfAkblGyurBG8ohzlo6tqXc= 2023-09-29T23:09:24.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x+UcKfAkblGyurBG8ohzlo6tqXc=', 'Content-Length': '0'} 2023-09-29T23:09:24.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b45c1aee446a9aa2-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.594 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-60 2023-09-29T23:09:24.595 DEBUG:boto:path=/dmuhsu-60/ 2023-09-29T23:09:24.595 DEBUG:boto:auth_path=/dmuhsu-60/ 2023-09-29T23:09:24.595 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.595 DEBUG:boto:Path: /dmuhsu-60/ 2023-09-29T23:09:24.595 DEBUG:boto:Data: 2023-09-29T23:09:24.596 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.596 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.596 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.596 DEBUG:boto:Params: {} 2023-09-29T23:09:24.597 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:09:24.597 DEBUG:boto:Token: None 2023-09-29T23:09:24.597 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/ 2023-09-29T23:09:24.597 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mWu6sbyd0EsL4FrtWHy6RL/gAEo= 2023-09-29T23:09:24.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mWu6sbyd0EsL4FrtWHy6RL/gAEo=', 'Content-Length': '0'} 2023-09-29T23:09:24.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6f5652475b3b2e8-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.647 DEBUG:boto:path=/dmuhsu-58/ 2023-09-29T23:09:24.647 DEBUG:boto:auth_path=/dmuhsu-58/ 2023-09-29T23:09:24.648 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.648 DEBUG:boto:Path: /dmuhsu-58/ 2023-09-29T23:09:24.648 DEBUG:boto:Data: 2023-09-29T23:09:24.648 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.648 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.648 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.648 DEBUG:boto:Params: {} 2023-09-29T23:09:24.648 DEBUG:boto:Token: None 2023-09-29T23:09:24.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/ 2023-09-29T23:09:24.648 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-29T23:09:24.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-29T23:09:24.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078a3afe3a25447e5-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.651 DEBUG:boto:path=/dmuhsu-58/obj1 2023-09-29T23:09:24.651 DEBUG:boto:auth_path=/dmuhsu-58/obj1 2023-09-29T23:09:24.651 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.651 DEBUG:boto:Path: /dmuhsu-58/obj1 2023-09-29T23:09:24.651 DEBUG:boto:Data: 2023-09-29T23:09:24.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.652 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.652 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.652 DEBUG:boto:Params: {} 2023-09-29T23:09:24.652 DEBUG:boto:Token: None 2023-09-29T23:09:24.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/obj1 2023-09-29T23:09:24.652 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r3eSGaBXvaxcCqSu3kjadoxhMKY= 2023-09-29T23:09:24.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3eSGaBXvaxcCqSu3kjadoxhMKY='} 2023-09-29T23:09:24.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aeaf2e5c237aecf-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.667 DEBUG:boto:path=/dmuhsu-58/ 2023-09-29T23:09:24.667 DEBUG:boto:auth_path=/dmuhsu-58/ 2023-09-29T23:09:24.667 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.667 DEBUG:boto:Path: /dmuhsu-58/ 2023-09-29T23:09:24.667 DEBUG:boto:Data: 2023-09-29T23:09:24.667 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.667 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.667 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.667 DEBUG:boto:Params: {} 2023-09-29T23:09:24.668 DEBUG:boto:Token: None 2023-09-29T23:09:24.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/ 2023-09-29T23:09:24.668 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-29T23:09:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-29T23:09:24.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001643d5fb261df2e-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.670 DEBUG:boto:path=/dmuhsu-58/obj2 2023-09-29T23:09:24.670 DEBUG:boto:auth_path=/dmuhsu-58/obj2 2023-09-29T23:09:24.671 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.671 DEBUG:boto:Path: /dmuhsu-58/obj2 2023-09-29T23:09:24.671 DEBUG:boto:Data: 2023-09-29T23:09:24.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.671 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.671 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.671 DEBUG:boto:Params: {} 2023-09-29T23:09:24.671 DEBUG:boto:Token: None 2023-09-29T23:09:24.671 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/obj2 2023-09-29T23:09:24.672 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a8GLC1smcbGcWBSFC73ir8IS+ao= 2023-09-29T23:09:24.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a8GLC1smcbGcWBSFC73ir8IS+ao='} 2023-09-29T23:09:24.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a31d48e522d1bae-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.679 DEBUG:boto:path=/dmuhsu-58/ 2023-09-29T23:09:24.679 DEBUG:boto:auth_path=/dmuhsu-58/ 2023-09-29T23:09:24.679 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.679 DEBUG:boto:Path: /dmuhsu-58/ 2023-09-29T23:09:24.680 DEBUG:boto:Data: 2023-09-29T23:09:24.680 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.680 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.680 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.680 DEBUG:boto:Params: {} 2023-09-29T23:09:24.680 DEBUG:boto:Token: None 2023-09-29T23:09:24.680 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/ 2023-09-29T23:09:24.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-29T23:09:24.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-29T23:09:24.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e99cd5e0c8f270b-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.682 DEBUG:boto:path=/dmuhsu-58/obj3 2023-09-29T23:09:24.683 DEBUG:boto:auth_path=/dmuhsu-58/obj3 2023-09-29T23:09:24.683 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.683 DEBUG:boto:Path: /dmuhsu-58/obj3 2023-09-29T23:09:24.683 DEBUG:boto:Data: 2023-09-29T23:09:24.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.683 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.684 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.684 DEBUG:boto:Params: {} 2023-09-29T23:09:24.684 DEBUG:boto:Token: None 2023-09-29T23:09:24.684 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/obj3 2023-09-29T23:09:24.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RvQNOm0M7YgjsokJStZToDKPB5E= 2023-09-29T23:09:24.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RvQNOm0M7YgjsokJStZToDKPB5E='} 2023-09-29T23:09:24.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036b059868cca6fc5-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.731 DEBUG:boto:path=/dmuhsu-58/ 2023-09-29T23:09:24.731 DEBUG:boto:auth_path=/dmuhsu-58/ 2023-09-29T23:09:24.731 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.731 DEBUG:boto:Path: /dmuhsu-58/ 2023-09-29T23:09:24.731 DEBUG:boto:Data: 2023-09-29T23:09:24.731 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.731 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.732 DEBUG:boto:Params: {} 2023-09-29T23:09:24.732 DEBUG:boto:Token: None 2023-09-29T23:09:24.732 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/ 2023-09-29T23:09:24.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-29T23:09:24.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-29T23:09:24.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a930750f33eb3e11-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.735 DEBUG:boto:path=/dmuhsu-58/obj4 2023-09-29T23:09:24.735 DEBUG:boto:auth_path=/dmuhsu-58/obj4 2023-09-29T23:09:24.735 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.735 DEBUG:boto:Path: /dmuhsu-58/obj4 2023-09-29T23:09:24.735 DEBUG:boto:Data: 2023-09-29T23:09:24.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:09:24.735 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.735 DEBUG:boto:Params: {} 2023-09-29T23:09:24.736 DEBUG:boto:Token: None 2023-09-29T23:09:24.736 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-58/obj4 2023-09-29T23:09:24.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zrQoELkuZ86CrPILwEDPQAYSrsE= 2023-09-29T23:09:24.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zrQoELkuZ86CrPILwEDPQAYSrsE='} 2023-09-29T23:09:24.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f52c31c77c4347c9-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.742 DEBUG:boto:path=/dmuhsu-59/ 2023-09-29T23:09:24.742 DEBUG:boto:auth_path=/dmuhsu-59/ 2023-09-29T23:09:24.742 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.742 DEBUG:boto:Path: /dmuhsu-59/ 2023-09-29T23:09:24.742 DEBUG:boto:Data: 2023-09-29T23:09:24.742 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.742 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.743 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.743 DEBUG:boto:Params: {} 2023-09-29T23:09:24.743 DEBUG:boto:Token: None 2023-09-29T23:09:24.743 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/ 2023-09-29T23:09:24.743 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-29T23:09:24.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-29T23:09:24.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000975f2911f3887c03-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.746 DEBUG:boto:path=/dmuhsu-59/obj1 2023-09-29T23:09:24.746 DEBUG:boto:auth_path=/dmuhsu-59/obj1 2023-09-29T23:09:24.746 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.746 DEBUG:boto:Path: /dmuhsu-59/obj1 2023-09-29T23:09:24.746 DEBUG:boto:Data: 2023-09-29T23:09:24.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.746 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.747 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.747 DEBUG:boto:Params: {} 2023-09-29T23:09:24.747 DEBUG:boto:Token: None 2023-09-29T23:09:24.747 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/obj1 2023-09-29T23:09:24.747 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CkcqC3FgC1E3RYvIQDlLWcauQp8= 2023-09-29T23:09:24.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CkcqC3FgC1E3RYvIQDlLWcauQp8='} 2023-09-29T23:09:24.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eb515244a2b9443-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.753 DEBUG:boto:path=/dmuhsu-59/ 2023-09-29T23:09:24.753 DEBUG:boto:auth_path=/dmuhsu-59/ 2023-09-29T23:09:24.753 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.754 DEBUG:boto:Path: /dmuhsu-59/ 2023-09-29T23:09:24.754 DEBUG:boto:Data: 2023-09-29T23:09:24.754 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.754 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.754 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.754 DEBUG:boto:Params: {} 2023-09-29T23:09:24.754 DEBUG:boto:Token: None 2023-09-29T23:09:24.754 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/ 2023-09-29T23:09:24.755 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-29T23:09:24.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-29T23:09:24.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc43aaf64c32230-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.757 DEBUG:boto:path=/dmuhsu-59/obj2 2023-09-29T23:09:24.757 DEBUG:boto:auth_path=/dmuhsu-59/obj2 2023-09-29T23:09:24.757 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.757 DEBUG:boto:Path: /dmuhsu-59/obj2 2023-09-29T23:09:24.757 DEBUG:boto:Data: 2023-09-29T23:09:24.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.758 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.758 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.758 DEBUG:boto:Params: {} 2023-09-29T23:09:24.758 DEBUG:boto:Token: None 2023-09-29T23:09:24.758 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/obj2 2023-09-29T23:09:24.758 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tOaec1XFw1e4xWc9jMPGXCVg/iE= 2023-09-29T23:09:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOaec1XFw1e4xWc9jMPGXCVg/iE='} 2023-09-29T23:09:24.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041cd03506df62f36-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.764 DEBUG:boto:path=/dmuhsu-59/ 2023-09-29T23:09:24.764 DEBUG:boto:auth_path=/dmuhsu-59/ 2023-09-29T23:09:24.764 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.764 DEBUG:boto:Path: /dmuhsu-59/ 2023-09-29T23:09:24.764 DEBUG:boto:Data: 2023-09-29T23:09:24.764 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.764 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.765 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.765 DEBUG:boto:Params: {} 2023-09-29T23:09:24.765 DEBUG:boto:Token: None 2023-09-29T23:09:24.765 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/ 2023-09-29T23:09:24.765 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-29T23:09:24.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-29T23:09:24.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000226bd23c1fc01ff2-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.768 DEBUG:boto:path=/dmuhsu-59/obj3 2023-09-29T23:09:24.768 DEBUG:boto:auth_path=/dmuhsu-59/obj3 2023-09-29T23:09:24.768 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.768 DEBUG:boto:Path: /dmuhsu-59/obj3 2023-09-29T23:09:24.768 DEBUG:boto:Data: 2023-09-29T23:09:24.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.768 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.768 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.768 DEBUG:boto:Params: {} 2023-09-29T23:09:24.769 DEBUG:boto:Token: None 2023-09-29T23:09:24.769 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/obj3 2023-09-29T23:09:24.769 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LXY8CWB7PlIo8j/ZyR3icLTEyjU= 2023-09-29T23:09:24.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LXY8CWB7PlIo8j/ZyR3icLTEyjU='} 2023-09-29T23:09:24.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010bf379230b61907-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.815 DEBUG:boto:path=/dmuhsu-59/ 2023-09-29T23:09:24.815 DEBUG:boto:auth_path=/dmuhsu-59/ 2023-09-29T23:09:24.815 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.815 DEBUG:boto:Path: /dmuhsu-59/ 2023-09-29T23:09:24.815 DEBUG:boto:Data: 2023-09-29T23:09:24.815 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.816 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.816 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.816 DEBUG:boto:Params: {} 2023-09-29T23:09:24.816 DEBUG:boto:Token: None 2023-09-29T23:09:24.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/ 2023-09-29T23:09:24.816 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-29T23:09:24.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-29T23:09:24.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad4c8b404398092-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.819 DEBUG:boto:path=/dmuhsu-59/obj4 2023-09-29T23:09:24.819 DEBUG:boto:auth_path=/dmuhsu-59/obj4 2023-09-29T23:09:24.819 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.819 DEBUG:boto:Path: /dmuhsu-59/obj4 2023-09-29T23:09:24.819 DEBUG:boto:Data: 2023-09-29T23:09:24.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.820 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:09:24.820 DEBUG:boto:Port: 8000 2023-09-29T23:09:24.820 DEBUG:boto:Params: {} 2023-09-29T23:09:24.820 DEBUG:boto:Token: None 2023-09-29T23:09:24.820 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-59/obj4 2023-09-29T23:09:24.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:b4VPeuxXjfryWjp5qadzlwPo/Uc= 2023-09-29T23:09:24.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b4VPeuxXjfryWjp5qadzlwPo/Uc='} 2023-09-29T23:09:24.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084b7f2aafe5bfa1b-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.826 DEBUG:boto:path=/dmuhsu-60/ 2023-09-29T23:09:24.826 DEBUG:boto:auth_path=/dmuhsu-60/ 2023-09-29T23:09:24.826 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.826 DEBUG:boto:Path: /dmuhsu-60/ 2023-09-29T23:09:24.827 DEBUG:boto:Data: 2023-09-29T23:09:24.827 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.827 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.827 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.827 DEBUG:boto:Params: {} 2023-09-29T23:09:24.827 DEBUG:boto:Token: None 2023-09-29T23:09:24.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/ 2023-09-29T23:09:24.827 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-29T23:09:24.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-29T23:09:24.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f6b8fa2ff99462d-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.830 DEBUG:boto:path=/dmuhsu-60/obj1 2023-09-29T23:09:24.830 DEBUG:boto:auth_path=/dmuhsu-60/obj1 2023-09-29T23:09:24.830 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.830 DEBUG:boto:Path: /dmuhsu-60/obj1 2023-09-29T23:09:24.831 DEBUG:boto:Data: 2023-09-29T23:09:24.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.831 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.831 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.831 DEBUG:boto:Params: {} 2023-09-29T23:09:24.831 DEBUG:boto:Token: None 2023-09-29T23:09:24.831 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/obj1 2023-09-29T23:09:24.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NU4ilN00NC9XqjFMn5k/VU8tL0Y= 2023-09-29T23:09:24.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NU4ilN00NC9XqjFMn5k/VU8tL0Y='} 2023-09-29T23:09:24.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046b956d5c903a95a-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.838 DEBUG:boto:path=/dmuhsu-60/ 2023-09-29T23:09:24.838 DEBUG:boto:auth_path=/dmuhsu-60/ 2023-09-29T23:09:24.838 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.838 DEBUG:boto:Path: /dmuhsu-60/ 2023-09-29T23:09:24.838 DEBUG:boto:Data: 2023-09-29T23:09:24.838 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.838 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.838 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.838 DEBUG:boto:Params: {} 2023-09-29T23:09:24.839 DEBUG:boto:Token: None 2023-09-29T23:09:24.839 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/ 2023-09-29T23:09:24.839 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-29T23:09:24.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-29T23:09:24.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fa3614f60078a3c-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.841 DEBUG:boto:path=/dmuhsu-60/obj2 2023-09-29T23:09:24.841 DEBUG:boto:auth_path=/dmuhsu-60/obj2 2023-09-29T23:09:24.841 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.841 DEBUG:boto:Path: /dmuhsu-60/obj2 2023-09-29T23:09:24.841 DEBUG:boto:Data: 2023-09-29T23:09:24.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.842 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.842 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.842 DEBUG:boto:Params: {} 2023-09-29T23:09:24.842 DEBUG:boto:Token: None 2023-09-29T23:09:24.842 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/obj2 2023-09-29T23:09:24.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nAsjm3ICjOcXOru7TMbPr/xzOFY= 2023-09-29T23:09:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nAsjm3ICjOcXOru7TMbPr/xzOFY='} 2023-09-29T23:09:24.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c96be487ef250668-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.848 DEBUG:boto:path=/dmuhsu-60/ 2023-09-29T23:09:24.849 DEBUG:boto:auth_path=/dmuhsu-60/ 2023-09-29T23:09:24.849 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.849 DEBUG:boto:Path: /dmuhsu-60/ 2023-09-29T23:09:24.849 DEBUG:boto:Data: 2023-09-29T23:09:24.849 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.849 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.849 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.849 DEBUG:boto:Params: {} 2023-09-29T23:09:24.850 DEBUG:boto:Token: None 2023-09-29T23:09:24.850 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/ 2023-09-29T23:09:24.850 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-29T23:09:24.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-29T23:09:24.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c710f255370593a-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.852 DEBUG:boto:path=/dmuhsu-60/obj3 2023-09-29T23:09:24.852 DEBUG:boto:auth_path=/dmuhsu-60/obj3 2023-09-29T23:09:24.852 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.852 DEBUG:boto:Path: /dmuhsu-60/obj3 2023-09-29T23:09:24.853 DEBUG:boto:Data: 2023-09-29T23:09:24.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.853 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.853 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.853 DEBUG:boto:Params: {} 2023-09-29T23:09:24.853 DEBUG:boto:Token: None 2023-09-29T23:09:24.853 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/obj3 2023-09-29T23:09:24.854 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hjJFs7V2TqGg/0ovR+YoJvlcCQ0= 2023-09-29T23:09:24.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hjJFs7V2TqGg/0ovR+YoJvlcCQ0='} 2023-09-29T23:09:24.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e650479d27ddc8c2-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.860 DEBUG:boto:path=/dmuhsu-60/ 2023-09-29T23:09:24.860 DEBUG:boto:auth_path=/dmuhsu-60/ 2023-09-29T23:09:24.860 DEBUG:boto:Method: HEAD 2023-09-29T23:09:24.860 DEBUG:boto:Path: /dmuhsu-60/ 2023-09-29T23:09:24.860 DEBUG:boto:Data: 2023-09-29T23:09:24.860 DEBUG:boto:Headers: {} 2023-09-29T23:09:24.860 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.860 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.861 DEBUG:boto:Params: {} 2023-09-29T23:09:24.861 DEBUG:boto:Token: None 2023-09-29T23:09:24.861 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/ 2023-09-29T23:09:24.861 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-29T23:09:24.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-29T23:09:24.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9fc81435926aa28-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.863 DEBUG:boto:path=/dmuhsu-60/obj4 2023-09-29T23:09:24.864 DEBUG:boto:auth_path=/dmuhsu-60/obj4 2023-09-29T23:09:24.864 DEBUG:boto:Method: PUT 2023-09-29T23:09:24.864 DEBUG:boto:Path: /dmuhsu-60/obj4 2023-09-29T23:09:24.864 DEBUG:boto:Data: 2023-09-29T23:09:24.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:09:24.864 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:09:24.864 DEBUG:boto:Port: 8001 2023-09-29T23:09:24.864 DEBUG:boto:Params: {} 2023-09-29T23:09:24.865 DEBUG:boto:Token: None 2023-09-29T23:09:24.865 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:09:24 GMT /dmuhsu-60/obj4 2023-09-29T23:09:24.865 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+QPrtg24KIPAGnxwp2M8314l244= 2023-09-29T23:09:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+QPrtg24KIPAGnxwp2M8314l244='} 2023-09-29T23:09:24.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c3997396109eac9-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:09:24.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:24.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:24.907 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:09:24.965 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:24.965 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:24.998 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:09:25.023 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:25.024 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028964.627375_480.1", 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:09:24.627375Z" 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:25.025 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:09:25.026 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:25.026 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:09:25.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:25.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:25.027 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:09:25.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:25.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:25.106 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:09:25.106+0000 7f41cd728880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:25.107 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:09:25.107+0000 7f41cd728880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:25.120 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:09:25.120 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:09:25.120 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:25.121 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:09:25.122 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028964.621572_479.1", 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:25.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:09:24.621572Z", 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:25.125 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:09:25.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:09:25Z" 2023-09-29T23:09:25.127 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:09:25.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028964.621572_479.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:09:24.621572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:09:25Z" } 2023-09-29T23:09:25.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.621572_479.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.621572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:09:25.129 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:25.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.621572_479.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:25.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696028964.627375_480.1 target=1_1696028964.621572_479.1 2023-09-29T23:09:30.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:30.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:30.130 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:09:30.183 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:30.184 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:30.216 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:09:30.216+0000 7f72d4322880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:30.217 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:09:30.217+0000 7f72d4322880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:30.231 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:09:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696028964.627375_480.1", 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-29T23:09:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:09:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:09:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:09:30Z" 2023-09-29T23:09:30.238 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:09:30.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028964.627375_480.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:09:24.627375Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:09:30Z" } 2023-09-29T23:09:30.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.627375_480.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.627375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:09:30.239 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:30.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:30.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:09:30.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:09:30.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:30.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:09:30.240 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:09:30.287 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:30.288 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:30.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:30.328+0000 7f9d1e327880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:30.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:30.329+0000 7f9d1e327880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:09:30.345 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:09:30.346 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:09:30.347 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:09:30.349 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:09:30.349 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:09:30.349 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:09:30.349 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696028964.627375_480.1", 2023-09-29T23:09:30.350 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:09:30.351 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:09:30.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:09:30Z" 2023-09-29T23:09:30.353 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:09:30.354 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696028964.627375_480.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:09:24.627375Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:09:30Z" } 2023-09-29T23:09:30.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.627375_480.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.627375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:09:30.355 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:30.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:09:30.355 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:09:30.355 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-58 2023-09-29T23:09:30.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.355 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:09:30.409 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:30.409 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:30.440 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-58 2023-09-29T23:09:30.454 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-58 2023-09-29T23:09:30.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.455 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:09:30.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:30.508 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:30.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:30.560+0000 7ff22809d880 1 bucket sync caught up with source: 2023-09-29T23:09:30.561 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [00000000001.24.6, , 00000000001.33.6, , 00000000001.25.6, , , 00000000001.28.6, , , ] 2023-09-29T23:09:30.561 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [00000000001.24.6, , 00000000001.33.6, , 00000000001.25.6, , , 00000000001.28.6, , , ] 2023-09-29T23:09:30.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:30.560+0000 7ff22809d880 0 bucket checkpoint complete 2023-09-29T23:09:30.584 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-58 2023-09-29T23:09:30.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:09:30.585 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:09:30.636 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:09:30.636 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:09:30.681 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:30.680+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:09:35.682 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:35.682+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:09:40.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:40.682+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:09:45.683 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:45.683+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:50.684+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:09:55.684 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:09:55.684+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:00.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:00.685+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:05.685 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:05.685+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:10.686 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:10.686+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:15.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:15.687+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:20.687 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:20.686+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:25.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:25.687+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:30.688 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:30.688+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:35.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:35.689+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:40.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:40.690+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:45.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:45.690+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:50.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:50.691+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:10:55.691 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:10:55.691+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:00.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:00.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:05.692 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:05.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:10.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:10.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:15.693 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:15.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:20.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:20.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:25.694 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:25.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:30.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:30.695+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:35.695 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:35.696+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:40.696 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:40.696+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:45.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:45.697+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:50.697 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:50.697+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:11:55.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:11:55.698+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:00.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:00.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:05.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:05.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:10.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:10.700+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:15.700 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:15.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:20.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:20.700+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:25.701 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:25.701+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:30.702 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:30.702+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:35.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:35.703+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:40.703 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:40.703+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:45.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:45.704+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:50.704 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:50.704+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:12:55.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:12:55.705+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:00.705 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:00.705+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:05.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:05.706+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:10.706 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:10.707+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:15.707 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:15.706+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:20.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:20.707+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:25.708 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:25.708+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:30.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:30.709+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:35.709 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:35.709+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:40.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:40.710+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:45.710 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:45.710+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:50.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:50.711+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:13:55.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:13:55.711+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:00.712 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:00.712+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:05.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:05.713+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:10.713 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:10.712+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:15.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:15.713+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:20.714 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:20.714+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-29T23:14:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:25.714+0000 7f556b926880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:14:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:25.714+0000 7f556b926880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:14:25.715 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:25.714+0000 7f556b926880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:14:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:14:25.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2023-09-29T23:14:25.735 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-61 2023-09-29T23:14:25.735 DEBUG:boto:path=/dmuhsu-61/ 2023-09-29T23:14:25.735 DEBUG:boto:auth_path=/dmuhsu-61/ 2023-09-29T23:14:25.735 DEBUG:boto:Method: PUT 2023-09-29T23:14:25.735 DEBUG:boto:Path: /dmuhsu-61/ 2023-09-29T23:14:25.736 DEBUG:boto:Data: 2023-09-29T23:14:25.736 DEBUG:boto:Headers: {} 2023-09-29T23:14:25.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:25.736 DEBUG:boto:Port: 8000 2023-09-29T23:14:25.736 DEBUG:boto:Params: {} 2023-09-29T23:14:25.736 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:14:25.736 DEBUG:boto:Token: None 2023-09-29T23:14:25.737 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:14:25 GMT /dmuhsu-61/ 2023-09-29T23:14:25.737 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O7Eb8B+aRqqkqraTx51k4tdHNTg= 2023-09-29T23:14:25.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O7Eb8B+aRqqkqraTx51k4tdHNTg=', 'Content-Length': '0'} 2023-09-29T23:14:25.766 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c7c070353728908-0065175a51-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:25.766 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-62 2023-09-29T23:14:25.767 DEBUG:boto:path=/dmuhsu-62/ 2023-09-29T23:14:25.767 DEBUG:boto:auth_path=/dmuhsu-62/ 2023-09-29T23:14:25.767 DEBUG:boto:Method: PUT 2023-09-29T23:14:25.767 DEBUG:boto:Path: /dmuhsu-62/ 2023-09-29T23:14:25.767 DEBUG:boto:Data: 2023-09-29T23:14:25.767 DEBUG:boto:Headers: {} 2023-09-29T23:14:25.767 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:14:25.767 DEBUG:boto:Port: 8000 2023-09-29T23:14:25.767 DEBUG:boto:Params: {} 2023-09-29T23:14:25.768 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:14:25.768 DEBUG:boto:Token: None 2023-09-29T23:14:25.768 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:14:25 GMT /dmuhsu-62/ 2023-09-29T23:14:25.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:80Grp2gCLGTnu92nwIHbVpxTxSE= 2023-09-29T23:14:25.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:80Grp2gCLGTnu92nwIHbVpxTxSE=', 'Content-Length': '0'} 2023-09-29T23:14:25.817 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c79aca77685d0a46-0065175a51-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:25.817 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=dmuhsu-63 2023-09-29T23:14:25.817 DEBUG:boto:path=/dmuhsu-63/ 2023-09-29T23:14:25.817 DEBUG:boto:auth_path=/dmuhsu-63/ 2023-09-29T23:14:25.817 DEBUG:boto:Method: PUT 2023-09-29T23:14:25.818 DEBUG:boto:Path: /dmuhsu-63/ 2023-09-29T23:14:25.818 DEBUG:boto:Data: 2023-09-29T23:14:25.818 DEBUG:boto:Headers: {} 2023-09-29T23:14:25.818 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:14:25.818 DEBUG:boto:Port: 8001 2023-09-29T23:14:25.818 DEBUG:boto:Params: {} 2023-09-29T23:14:25.818 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:14:25.818 DEBUG:boto:Token: None 2023-09-29T23:14:25.819 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:14:25 GMT /dmuhsu-63/ 2023-09-29T23:14:25.819 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mt6wq04OFnPbU+9JPmubOS7ejdE= 2023-09-29T23:14:25.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mt6wq04OFnPbU+9JPmubOS7ejdE=', 'Content-Length': '0'} 2023-09-29T23:14:25.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007965a2f919c84bbd-0065175a51-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:25.868 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:25.868 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:25.868 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploads 2023-09-29T23:14:25.868 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploads 2023-09-29T23:14:25.868 DEBUG:boto:Method: POST 2023-09-29T23:14:25.868 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploads 2023-09-29T23:14:25.868 DEBUG:boto:Data: 2023-09-29T23:14:25.868 DEBUG:boto:Headers: {} 2023-09-29T23:14:25.869 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:25.869 DEBUG:boto:Port: 8000 2023-09-29T23:14:25.869 DEBUG:boto:Params: {} 2023-09-29T23:14:25.869 DEBUG:boto:Token: None 2023-09-29T23:14:25.869 DEBUG:boto:StringToSign: POST Fri, 29 Sep 2023 23:14:25 GMT /dmuhsu-61/MULTIPART?uploads 2023-09-29T23:14:25.869 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XY9wXvnG02UHqB2kas3vG1yC/EE= 2023-09-29T23:14:25.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XY9wXvnG02UHqB2kas3vG1yC/EE=', 'Content-Length': '0'} 2023-09-29T23:14:27.805 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085707f87a3c98815-0065175a51-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:27.805 DEBUG:boto:b'dmuhsu-61MULTIPART2~Bta-4Ct08R581PcClvc_t-Hrubf8obn' 2023-09-29T23:14:27.838 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.838 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.838 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-29T23:14:27.838 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-29T23:14:27.838 DEBUG:boto:Method: PUT 2023-09-29T23:14:27.839 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-29T23:14:27.839 DEBUG:boto:Data: 2023-09-29T23:14:27.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T23:14:27.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:27.839 DEBUG:boto:Port: 8000 2023-09-29T23:14:27.839 DEBUG:boto:Params: {} 2023-09-29T23:14:27.839 DEBUG:boto:Token: None 2023-09-29T23:14:27.839 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 29 Sep 2023 23:14:27 GMT /dmuhsu-61/MULTIPART?partNumber=1&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:27.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:IRcPN6LH/QV08ePjdOm0lWQAuqw= 2023-09-29T23:14:27.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IRcPN6LH/QV08ePjdOm0lWQAuqw='} 2023-09-29T23:14:27.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c2aed21dbc1b2d7-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:27.918 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.918 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.919 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-29T23:14:27.919 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-29T23:14:27.919 DEBUG:boto:Method: PUT 2023-09-29T23:14:27.919 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-29T23:14:27.919 DEBUG:boto:Data: 2023-09-29T23:14:27.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T23:14:27.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:27.919 DEBUG:boto:Port: 8000 2023-09-29T23:14:27.919 DEBUG:boto:Params: {} 2023-09-29T23:14:27.920 DEBUG:boto:Token: None 2023-09-29T23:14:27.920 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 29 Sep 2023 23:14:27 GMT /dmuhsu-61/MULTIPART?partNumber=2&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:27.920 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hY0NAs+DaLCWRL6DE5uhTnbyH1o= 2023-09-29T23:14:27.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hY0NAs+DaLCWRL6DE5uhTnbyH1o='} 2023-09-29T23:14:27.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082c20d3318db3b68-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:27.971 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.971 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:27.971 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-29T23:14:27.971 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-29T23:14:27.971 DEBUG:boto:Method: PUT 2023-09-29T23:14:27.971 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-29T23:14:27.971 DEBUG:boto:Data: 2023-09-29T23:14:27.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T23:14:27.972 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:27.972 DEBUG:boto:Port: 8000 2023-09-29T23:14:27.972 DEBUG:boto:Params: {} 2023-09-29T23:14:27.972 DEBUG:boto:Token: None 2023-09-29T23:14:27.972 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 29 Sep 2023 23:14:27 GMT /dmuhsu-61/MULTIPART?partNumber=3&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:27.972 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PFYoSQCueu3zY5v6eSVVcIfHq38= 2023-09-29T23:14:27.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PFYoSQCueu3zY5v6eSVVcIfHq38='} 2023-09-29T23:14:28.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001433115c78a5b09b-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:28.024 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.025 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.025 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-29T23:14:28.025 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-29T23:14:28.025 DEBUG:boto:Method: PUT 2023-09-29T23:14:28.025 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-29T23:14:28.025 DEBUG:boto:Data: 2023-09-29T23:14:28.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T23:14:28.025 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:28.025 DEBUG:boto:Port: 8000 2023-09-29T23:14:28.026 DEBUG:boto:Params: {} 2023-09-29T23:14:28.026 DEBUG:boto:Token: None 2023-09-29T23:14:28.026 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 29 Sep 2023 23:14:28 GMT /dmuhsu-61/MULTIPART?partNumber=4&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.026 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FVNFRTDnW72fRPCR3IHUqtafLgQ= 2023-09-29T23:14:28.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVNFRTDnW72fRPCR3IHUqtafLgQ='} 2023-09-29T23:14:28.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b644bb9ddbfc82a-0065175a54-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:28.056 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.057 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.057 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.057 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.057 DEBUG:boto:Method: GET 2023-09-29T23:14:28.057 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.057 DEBUG:boto:Data: 2023-09-29T23:14:28.057 DEBUG:boto:Headers: {} 2023-09-29T23:14:28.057 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:28.058 DEBUG:boto:Port: 8000 2023-09-29T23:14:28.058 DEBUG:boto:Params: {} 2023-09-29T23:14:28.058 DEBUG:boto:Token: None 2023-09-29T23:14:28.058 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:14:28 GMT /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.058 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:p9Fj/tsOo6C1m2HW4WpvEtWtkOE= 2023-09-29T23:14:28.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p9Fj/tsOo6C1m2HW4WpvEtWtkOE=', 'Content-Length': '0'} 2023-09-29T23:14:28.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab2caed15863c21-0065175a54-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:28.107 DEBUG:boto:path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.107 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART 2023-09-29T23:14:28.107 DEBUG:boto:path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.107 DEBUG:boto:auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.107 DEBUG:boto:Method: POST 2023-09-29T23:14:28.107 DEBUG:boto:Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.108 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-09-29T23:14:28.108 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-09-29T23:14:28.108 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:14:28.108 DEBUG:boto:Port: 8000 2023-09-29T23:14:28.108 DEBUG:boto:Params: {} 2023-09-29T23:14:28.108 DEBUG:boto:Token: None 2023-09-29T23:14:28.108 DEBUG:boto:StringToSign: POST text/xml Fri, 29 Sep 2023 23:14:28 GMT /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-29T23:14:28.109 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:B/9qoAhPI3gA5vLtsT44co2iqzk= 2023-09-29T23:14:28.109 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:B/9qoAhPI3gA5vLtsT44co2iqzk=', 'Content-Length': '460'} 2023-09-29T23:14:28.116 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f02c3b158f08dd34-0065175a54-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:14:28.117 DEBUG:boto:http://smithi175.front.sepia.ceph.com:8000/dmuhsu-61/MULTIPARTdmuhsu-61MULTIPART 2023-09-29T23:14:28.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.118 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:28.174 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:28.174 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:28.213 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:14:28.228 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:14:28.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.228 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:14:28.228 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:14:28.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029265.852881_493.1", 2023-09-29T23:14:28.229 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:14:25.852881Z" 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.230 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:14:28.231 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:14:28.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:14:28.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.232 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:28.286 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:28.286 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:28.318 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:14:28.318+0000 7f8fd2013880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:14:28.319+0000 7f8fd2013880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:14:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:14:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:14:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:14:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696029265.852881_493.1", 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:14:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:14:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:14:28Z" 2023-09-29T23:14:28.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:14:28.340 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029265.852881_493.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:25.852881Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:28Z" } 2023-09-29T23:14:28.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029265.852881_493.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:25.852881Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:14:28.340 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:14:28.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:14:28.341 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:14:28.341 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:14:28.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:28.341 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:28.387 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:28.387 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:28.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:28.426+0000 7f4d53410880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:28.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:28.427+0000 7f4d53410880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:28.442 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:14:28.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:14:28.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:14:28.445 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029265.852881_493.1", 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:14:28.446 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:14:28.447 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.448 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:14:28.449 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:14:28.449 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:14:28.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:14:28.449 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:14:28Z" 2023-09-29T23:14:28.449 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:14:28.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029265.852881_493.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:25.852881Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:28Z" } 2023-09-29T23:14:28.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029265.852881_493.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:25.852881Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:14:28.450 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:14:28.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:14:28.450 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:14:28.451 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-61 2023-09-29T23:14:28.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:14:28.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:14:28.451 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:14:28.504 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:28.505 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:28.536 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-61 2023-09-29T23:14:28.550 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-61 2023-09-29T23:14:28.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:14:28.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:14:28.550 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:14:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:28.604 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:28.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:28.643+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:28.643 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:28.643 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:33.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:33.644+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:33.644 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:33.644 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:38.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:38.645+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:38.645 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:38.646 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:43.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:43.647+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:43.647 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:43.647 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:48.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:48.648+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:48.648 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:48.648 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:53.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:53.649+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:53.650 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:53.650 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:14:58.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:14:58.651+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:14:58.651 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:14:58.651 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:03.652 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:03.652+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:03.652 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:03.652 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:08.653 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:08.653+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:08.654 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:08.654 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:13.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:13.653+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:13.654 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:13.655 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:18.656 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:18.654+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:18.656 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:18.656 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:23.657 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:23.656+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:23.657 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:23.657 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:28.658 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:28.658+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-29T23:15:28.659 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-29T23:15:28.659 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:33.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:33.660+0000 7fe519c9d880 1 bucket sync caught up with source: 2023-09-29T23:15:33.661 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:33.662 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-29T23:15:33.662 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:33.660+0000 7fe519c9d880 0 bucket checkpoint complete 2023-09-29T23:15:33.679 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-61 2023-09-29T23:15:33.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:15:33.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:15:33.680 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:15:33.736 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:33.736 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:33.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:33.791+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:15:38.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:38.792+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:15:43.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:43.792+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:15:48.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:48.793+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:15:53.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:53.793+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:15:58.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:15:58.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:03.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:03.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:08.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:08.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:13.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:13.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:18.795+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:23.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:23.797+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:28.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:28.797+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:33.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:33.798+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:38.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:38.798+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:43.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:43.799+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:48.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:48.799+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:53.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:53.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:16:58.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:16:58.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:03.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:03.801+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:08.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:08.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:13.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:13.801+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:18.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:18.802+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:23.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:23.803+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:28.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:28.803+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:33.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:33.804+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:38.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:38.804+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:43.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:43.805+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:48.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:48.806+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:53.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:53.806+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:17:58.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:17:58.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:03.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:03.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:08.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:08.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:13.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:13.808+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:18.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:18.808+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:23.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:23.809+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:28.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:28.810+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:33.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:33.811+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:38.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:38.811+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:43.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:43.812+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:48.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:48.812+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:53.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:53.813+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:18:58.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:18:58.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:03.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:03.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:08.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:08.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:13.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:13.815+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:18.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:18.815+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:23.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:23.816+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:28.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:28.817+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:33.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:33.817+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:38.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:38.818+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:43.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:43.818+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:48.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:48.819+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:53.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:53.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:19:58.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:19:58.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:03.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:03.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:08.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:08.821+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:13.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:13.821+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:18.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:18.822+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:23.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:23.823+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-29T23:20:28.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:28.823+0000 7f797e926880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:20:28.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:28.823+0000 7f797e926880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:20:28.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:28.823+0000 7f797e926880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:20:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:20:28.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2023-09-29T23:20:28.843 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-64 2023-09-29T23:20:28.843 DEBUG:boto:path=/dmuhsu-64/ 2023-09-29T23:20:28.843 DEBUG:boto:auth_path=/dmuhsu-64/ 2023-09-29T23:20:28.844 DEBUG:boto:Method: PUT 2023-09-29T23:20:28.844 DEBUG:boto:Path: /dmuhsu-64/ 2023-09-29T23:20:28.844 DEBUG:boto:Data: 2023-09-29T23:20:28.844 DEBUG:boto:Headers: {} 2023-09-29T23:20:28.844 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:28.844 DEBUG:boto:Port: 8000 2023-09-29T23:20:28.844 DEBUG:boto:Params: {} 2023-09-29T23:20:28.844 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:20:28.845 DEBUG:boto:Token: None 2023-09-29T23:20:28.845 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:20:28 GMT /dmuhsu-64/ 2023-09-29T23:20:28.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NdSvXSTbG7bNJq18Cn+u/M0JYrU= 2023-09-29T23:20:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NdSvXSTbG7bNJq18Cn+u/M0JYrU=', 'Content-Length': '0'} 2023-09-29T23:20:28.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8a0cdde47d77a73-0065175bbc-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:28.865 DEBUG:boto:path=/dmuhsu-64/testobj-sse-c 2023-09-29T23:20:28.865 DEBUG:boto:auth_path=/dmuhsu-64/testobj-sse-c 2023-09-29T23:20:28.865 DEBUG:boto:Method: PUT 2023-09-29T23:20:28.865 DEBUG:boto:Path: /dmuhsu-64/testobj-sse-c 2023-09-29T23:20:28.865 DEBUG:boto:Data: 2023-09-29T23:20:28.866 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T23:20:28.866 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:28.866 DEBUG:boto:Port: 8000 2023-09-29T23:20:28.866 DEBUG:boto:Params: {} 2023-09-29T23:20:28.866 DEBUG:boto:Token: None 2023-09-29T23:20:28.866 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 23:20:28 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dmuhsu-64/testobj-sse-c 2023-09-29T23:20:28.866 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xBIVKScy6GZNH8IVvB7ESPmzeY8= 2023-09-29T23:20:28.867 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xBIVKScy6GZNH8IVvB7ESPmzeY8='} 2023-09-29T23:20:28.973 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000dc20ca3768f84cd3-0065175bbc-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:28.973 DEBUG:boto:path=/dmuhsu-64/testobj-sse-kms 2023-09-29T23:20:28.973 DEBUG:boto:auth_path=/dmuhsu-64/testobj-sse-kms 2023-09-29T23:20:28.974 DEBUG:boto:Method: PUT 2023-09-29T23:20:28.974 DEBUG:boto:Path: /dmuhsu-64/testobj-sse-kms 2023-09-29T23:20:28.974 DEBUG:boto:Data: 2023-09-29T23:20:28.974 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T23:20:28.974 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:28.974 DEBUG:boto:Port: 8000 2023-09-29T23:20:28.974 DEBUG:boto:Params: {} 2023-09-29T23:20:28.974 DEBUG:boto:Token: None 2023-09-29T23:20:28.975 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 23:20:28 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /dmuhsu-64/testobj-sse-kms 2023-09-29T23:20:28.975 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Zw7kcb/VXu50IZfheVzU7CR94xo= 2023-09-29T23:20:28.975 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Zw7kcb/VXu50IZfheVzU7CR94xo='} 2023-09-29T23:20:28.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZG11aHN1LTY0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000072751116fd97734b-0065175bbc-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:28.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:28.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:28.994 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.055 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.101 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:20:29.121 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:20:29.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.122 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029628.863421_509.1", 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:20:28.863421Z" 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.123 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:20:29.124 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.124 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:20:29.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.125 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.176 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.176 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.207 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:20:29.207+0000 7f7738b11880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.208 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:20:29.208+0000 7f7738b11880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.222 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:20:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:20:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:20:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:20:29.222 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696029628.863421_509.1", 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:20:29Z" 2023-09-29T23:20:29.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:20:29.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029628.863421_509.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:20:28.863421Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:29Z" } 2023-09-29T23:20:29.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029628.863421_509.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:28.863421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:20:29.230 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.230 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:20:29.230 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:20:29.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.230 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.278 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.278 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:29.332+0000 7f0fc3044880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:29.333+0000 7f0fc3044880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:20:29.358 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:20:29.361 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.362 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:20:29.363 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.364 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029628.863421_509.1", 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.365 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.366 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.367 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:20:29Z" 2023-09-29T23:20:29.368 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:20:29.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029628.863421_509.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:20:28.863421Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:29Z" } 2023-09-29T23:20:29.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029628.863421_509.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:28.863421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:20:29.369 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:20:29.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:29.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:29.370 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:20:29.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.453 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-64 2023-09-29T23:20:29.467 DEBUG:boto:path=/dmuhsu-64/ 2023-09-29T23:20:29.467 DEBUG:boto:auth_path=/dmuhsu-64/ 2023-09-29T23:20:29.467 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.468 DEBUG:boto:Path: /dmuhsu-64/ 2023-09-29T23:20:29.468 DEBUG:boto:Data: 2023-09-29T23:20:29.468 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.468 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:20:29.468 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.468 DEBUG:boto:Params: {} 2023-09-29T23:20:29.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:20:29.468 DEBUG:boto:Token: None 2023-09-29T23:20:29.469 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-64/ 2023-09-29T23:20:29.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TgQlr0B3/CQgKsZ0oDSUOr4jXxw= 2023-09-29T23:20:29.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TgQlr0B3/CQgKsZ0oDSUOr4jXxw=', 'Content-Length': '0'} 2023-09-29T23:20:29.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fe9d3f2010cce65-0065175bbd-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.472 DEBUG:boto:path=/dmuhsu-64/testobj-sse-c 2023-09-29T23:20:29.472 DEBUG:boto:auth_path=/dmuhsu-64/testobj-sse-c 2023-09-29T23:20:29.472 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.473 DEBUG:boto:Path: /dmuhsu-64/testobj-sse-c 2023-09-29T23:20:29.473 DEBUG:boto:Data: 2023-09-29T23:20:29.473 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-29T23:20:29.473 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:20:29.473 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.473 DEBUG:boto:Params: {} 2023-09-29T23:20:29.473 DEBUG:boto:Token: None 2023-09-29T23:20:29.473 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dmuhsu-64/testobj-sse-c 2023-09-29T23:20:29.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+PEj6Ra1+D/WbbwyopcPKn49tdk= 2023-09-29T23:20:29.474 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+PEj6Ra1+D/WbbwyopcPKn49tdk=', 'Content-Length': '0'} 2023-09-29T23:20:29.475 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000159a37d1a924aedd-0065175bbd-14ab-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2023-09-29T23:20:29.476 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-65 2023-09-29T23:20:29.476 DEBUG:boto:path=/dmuhsu-65/ 2023-09-29T23:20:29.476 DEBUG:boto:auth_path=/dmuhsu-65/ 2023-09-29T23:20:29.476 DEBUG:boto:Method: PUT 2023-09-29T23:20:29.476 DEBUG:boto:Path: /dmuhsu-65/ 2023-09-29T23:20:29.476 DEBUG:boto:Data: 2023-09-29T23:20:29.477 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.477 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.477 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.477 DEBUG:boto:Params: {} 2023-09-29T23:20:29.477 DEBUG:boto:Token: None 2023-09-29T23:20:29.477 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/ 2023-09-29T23:20:29.477 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ie+XQiUtDMkkXKcNuzcGV/VZy6g= 2023-09-29T23:20:29.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ie+XQiUtDMkkXKcNuzcGV/VZy6g=', 'Content-Length': '0'} 2023-09-29T23:20:29.504 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a1e33b54930e172b-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.504 DEBUG:boto:path=/dmuhsu-65/ 2023-09-29T23:20:29.504 DEBUG:boto:auth_path=/dmuhsu-65/ 2023-09-29T23:20:29.504 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.504 DEBUG:boto:Path: /dmuhsu-65/ 2023-09-29T23:20:29.504 DEBUG:boto:Data: 2023-09-29T23:20:29.505 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.505 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.505 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.505 DEBUG:boto:Params: {} 2023-09-29T23:20:29.505 DEBUG:boto:Token: None 2023-09-29T23:20:29.505 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/ 2023-09-29T23:20:29.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-29T23:20:29.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-29T23:20:29.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9e2a97e18443a76-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.508 DEBUG:boto:path=/dmuhsu-65/a 2023-09-29T23:20:29.508 DEBUG:boto:auth_path=/dmuhsu-65/a 2023-09-29T23:20:29.508 DEBUG:boto:Method: PUT 2023-09-29T23:20:29.508 DEBUG:boto:Path: /dmuhsu-65/a 2023-09-29T23:20:29.508 DEBUG:boto:Data: 2023-09-29T23:20:29.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:20:29.508 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.508 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.508 DEBUG:boto:Params: {} 2023-09-29T23:20:29.509 DEBUG:boto:Token: None 2023-09-29T23:20:29.509 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/a 2023-09-29T23:20:29.509 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9wFgG6E1nADkzOTX5XiQHthi+sg= 2023-09-29T23:20:29.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9wFgG6E1nADkzOTX5XiQHthi+sg='} 2023-09-29T23:20:29.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccebaeffd6b0f8af-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.555 DEBUG:boto:path=/dmuhsu-65/ 2023-09-29T23:20:29.555 DEBUG:boto:auth_path=/dmuhsu-65/ 2023-09-29T23:20:29.555 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.555 DEBUG:boto:Path: /dmuhsu-65/ 2023-09-29T23:20:29.555 DEBUG:boto:Data: 2023-09-29T23:20:29.555 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.555 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.556 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.556 DEBUG:boto:Params: {} 2023-09-29T23:20:29.556 DEBUG:boto:Token: None 2023-09-29T23:20:29.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/ 2023-09-29T23:20:29.556 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-29T23:20:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-29T23:20:29.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb2f8c66909a6a08-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.559 DEBUG:boto:path=/dmuhsu-65/b 2023-09-29T23:20:29.559 DEBUG:boto:auth_path=/dmuhsu-65/b 2023-09-29T23:20:29.559 DEBUG:boto:Method: PUT 2023-09-29T23:20:29.559 DEBUG:boto:Path: /dmuhsu-65/b 2023-09-29T23:20:29.559 DEBUG:boto:Data: 2023-09-29T23:20:29.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:20:29.560 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.560 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.560 DEBUG:boto:Params: {} 2023-09-29T23:20:29.560 DEBUG:boto:Token: None 2023-09-29T23:20:29.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/b 2023-09-29T23:20:29.560 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AkP8GMaXXlisDMeYWOkFd5d7aBE= 2023-09-29T23:20:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AkP8GMaXXlisDMeYWOkFd5d7aBE='} 2023-09-29T23:20:29.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab94224ce6620e71-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.569 DEBUG:boto:path=/dmuhsu-65/ 2023-09-29T23:20:29.569 DEBUG:boto:auth_path=/dmuhsu-65/ 2023-09-29T23:20:29.570 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.570 DEBUG:boto:Path: /dmuhsu-65/ 2023-09-29T23:20:29.570 DEBUG:boto:Data: 2023-09-29T23:20:29.570 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.570 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.570 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.570 DEBUG:boto:Params: {} 2023-09-29T23:20:29.570 DEBUG:boto:Token: None 2023-09-29T23:20:29.570 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/ 2023-09-29T23:20:29.571 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-29T23:20:29.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-29T23:20:29.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009066bd9b50bbc11c-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.573 DEBUG:boto:path=/dmuhsu-65/c 2023-09-29T23:20:29.573 DEBUG:boto:auth_path=/dmuhsu-65/c 2023-09-29T23:20:29.573 DEBUG:boto:Method: PUT 2023-09-29T23:20:29.573 DEBUG:boto:Path: /dmuhsu-65/c 2023-09-29T23:20:29.573 DEBUG:boto:Data: 2023-09-29T23:20:29.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:20:29.573 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.573 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.574 DEBUG:boto:Params: {} 2023-09-29T23:20:29.574 DEBUG:boto:Token: None 2023-09-29T23:20:29.574 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/c 2023-09-29T23:20:29.574 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zWFlj7WJ7MRN6LKxQxRI4M1NVzE= 2023-09-29T23:20:29.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zWFlj7WJ7MRN6LKxQxRI4M1NVzE='} 2023-09-29T23:20:29.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034727b71be43be0c-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.623 DEBUG:boto:path=/dmuhsu-65/ 2023-09-29T23:20:29.623 DEBUG:boto:auth_path=/dmuhsu-65/ 2023-09-29T23:20:29.623 DEBUG:boto:Method: HEAD 2023-09-29T23:20:29.623 DEBUG:boto:Path: /dmuhsu-65/ 2023-09-29T23:20:29.624 DEBUG:boto:Data: 2023-09-29T23:20:29.624 DEBUG:boto:Headers: {} 2023-09-29T23:20:29.624 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.624 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.624 DEBUG:boto:Params: {} 2023-09-29T23:20:29.625 DEBUG:boto:Token: None 2023-09-29T23:20:29.625 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/ 2023-09-29T23:20:29.625 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-29T23:20:29.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-29T23:20:29.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f482860d78577557-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.628 DEBUG:boto:path=/dmuhsu-65/d 2023-09-29T23:20:29.628 DEBUG:boto:auth_path=/dmuhsu-65/d 2023-09-29T23:20:29.628 DEBUG:boto:Method: PUT 2023-09-29T23:20:29.628 DEBUG:boto:Path: /dmuhsu-65/d 2023-09-29T23:20:29.628 DEBUG:boto:Data: 2023-09-29T23:20:29.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:20:29.628 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:20:29.628 DEBUG:boto:Port: 8000 2023-09-29T23:20:29.628 DEBUG:boto:Params: {} 2023-09-29T23:20:29.629 DEBUG:boto:Token: None 2023-09-29T23:20:29.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:20:29 GMT /dmuhsu-65/d 2023-09-29T23:20:29.629 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Kki+7iRBfqTN9cUc+OKemU8fXws= 2023-09-29T23:20:29.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kki+7iRBfqTN9cUc+OKemU8fXws='} 2023-09-29T23:20:29.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9d6f241f61d3d03-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:20:29.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.635 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.679 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.679 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.725 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029629.500136_513.1", 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:20:29.500136Z" 2023-09-29T23:20:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:20:29.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.745 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:20:29.746 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.746 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:20:29.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.746 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.792 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:20:29.821+0000 7f14dc49d880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.822 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:20:29.822+0000 7f14dc49d880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:20:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:20:29.835 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:20:29.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696029629.500136_513.1", 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.840 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:20:29.841 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:20:29.842 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:20:29.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:20:29.842 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:20:29Z" 2023-09-29T23:20:29.842 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:20:29.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029629.500136_513.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:20:29.500136Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:29Z" } 2023-09-29T23:20:29.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.500136_513.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.500136Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:20:29.843 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:20:29.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:20:29.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:29.844 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:29.889 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:29.889 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:29.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:29.929+0000 7f493e737880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.930 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:29.930+0000 7f493e737880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:29.943 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:20:29.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.945 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.946 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029629.490601_510.1", 2023-09-29T23:20:29.947 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:20:29.490601Z", 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:20:29.948 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:29.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:20:29Z" 2023-09-29T23:20:29.950 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:20:29.951 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029629.490601_510.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:20:29.490601Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:29Z" } 2023-09-29T23:20:29.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.490601_510.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.490601Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:20:29.952 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.490601_510.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:29.952 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696029629.500136_513.1 target=1_1696029629.490601_510.1 2023-09-29T23:20:34.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:34.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:34.954 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:35.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:35.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:35.052 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:35.052+0000 7f9a37526880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:35.053 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:35.053+0000 7f9a37526880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:35.073 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:20:35.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:35.075 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:20:35.075 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:35.075 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:35.075 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:20:35.075 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:35.076 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:35.077 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029629.500136_513.1", 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:20:35.078 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:20:35.079 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:20:35Z" 2023-09-29T23:20:35.080 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:20:35.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029629.500136_513.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:20:29.500136Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:35Z" } 2023-09-29T23:20:35.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.500136_513.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.500136Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:20:35.081 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:35.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:20:35.082 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:20:35.082 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-65 2023-09-29T23:20:35.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:35.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:35.082 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:20:35.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:35.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:35.176 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-65 2023-09-29T23:20:35.191 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-65 2023-09-29T23:20:35.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:35.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:35.191 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:20:35.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:35.246 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:35.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:35.293+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-29T23:20:40.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:40.293+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-29T23:20:45.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:45.294+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-29T23:20:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:50.296+0000 7fd426c15880 1 bucket sync caught up with source: 2023-09-29T23:20:50.296 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.14.6, , , , 00000000001.43.6, , , 00000000001.48.6, , 00000000001.43.6] 2023-09-29T23:20:50.296 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.14.6, , , , 00000000001.43.6, , , 00000000001.48.6, , 00000000001.43.6] 2023-09-29T23:20:50.296 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:50.296+0000 7fd426c15880 0 bucket checkpoint complete 2023-09-29T23:20:50.315 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-65 2023-09-29T23:20:50.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:50.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:20:50.316 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:20:50.372 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:50.372 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:50.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:50.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:20:55.416 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:20:55.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:00.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:00.417+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:05.417 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:05.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:10.418 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:10.417+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:15.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:15.418+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:20.419 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:20.419+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:25.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:25.420+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:30.420 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:30.421+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:35.421 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:35.421+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:40.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:40.422+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:45.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:45.423+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:50.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:50.423+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:21:55.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:21:55.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:00.424 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:00.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:05.425 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:05.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:10.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:10.425+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:15.426 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:15.425+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:20.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:20.427+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:25.427 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:25.427+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:30.428 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:30.428+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:35.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:35.429+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:40.429 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:40.429+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:45.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:45.430+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:50.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:50.430+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:22:55.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:22:55.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:00.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:00.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:05.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:05.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:10.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:10.432+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:15.433 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:15.432+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:20.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:20.434+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:25.434 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:25.434+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:30.435 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:30.435+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:35.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:35.436+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:40.436+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:45.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:45.437+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:50.437 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:50.437+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:23:55.438 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:23:55.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:00.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:00.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:05.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:05.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:10.439 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:10.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:15.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:15.440+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:20.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:20.441+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:25.441 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:25.441+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:30.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:30.442+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:35.442 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:35.442+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:40.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:40.443+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:45.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:45.443+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:50.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:50.444+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:24:55.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:24:55.445+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:00.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:00.444+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:05.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:05.445+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:10.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:10.446+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:15.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:15.447+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:20.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:20.447+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:25.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:25.448+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:30.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:30.448+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:35.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:35.449+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:40.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:40.450+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-29T23:25:45.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:45.450+0000 7f3710b29880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:25:45.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:45.450+0000 7f3710b29880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:25:45.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:45.450+0000 7f3710b29880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:25:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:25:45.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2023-09-29T23:25:45.473 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-66 2023-09-29T23:25:45.473 DEBUG:boto:path=/dmuhsu-66/ 2023-09-29T23:25:45.473 DEBUG:boto:auth_path=/dmuhsu-66/ 2023-09-29T23:25:45.474 DEBUG:boto:Method: PUT 2023-09-29T23:25:45.474 DEBUG:boto:Path: /dmuhsu-66/ 2023-09-29T23:25:45.474 DEBUG:boto:Data: 2023-09-29T23:25:45.474 DEBUG:boto:Headers: {} 2023-09-29T23:25:45.475 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.475 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.475 DEBUG:boto:Params: {} 2023-09-29T23:25:45.475 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:25:45.476 DEBUG:boto:Token: None 2023-09-29T23:25:45.476 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/ 2023-09-29T23:25:45.476 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GIpKm0hdZDG8KcSLCJaj3M7tY4Y= 2023-09-29T23:25:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIpKm0hdZDG8KcSLCJaj3M7tY4Y=', 'Content-Length': '0'} 2023-09-29T23:25:45.499 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c49a2a09b832d45-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.499 DEBUG:boto:path=/dmuhsu-66/ 2023-09-29T23:25:45.499 DEBUG:boto:auth_path=/dmuhsu-66/ 2023-09-29T23:25:45.500 DEBUG:boto:Method: HEAD 2023-09-29T23:25:45.500 DEBUG:boto:Path: /dmuhsu-66/ 2023-09-29T23:25:45.500 DEBUG:boto:Data: 2023-09-29T23:25:45.500 DEBUG:boto:Headers: {} 2023-09-29T23:25:45.500 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.501 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.501 DEBUG:boto:Params: {} 2023-09-29T23:25:45.501 DEBUG:boto:Token: None 2023-09-29T23:25:45.501 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/ 2023-09-29T23:25:45.502 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-29T23:25:45.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-29T23:25:45.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007addb06af55ece7-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.505 DEBUG:boto:path=/dmuhsu-66/a 2023-09-29T23:25:45.505 DEBUG:boto:auth_path=/dmuhsu-66/a 2023-09-29T23:25:45.506 DEBUG:boto:Method: PUT 2023-09-29T23:25:45.506 DEBUG:boto:Path: /dmuhsu-66/a 2023-09-29T23:25:45.506 DEBUG:boto:Data: 2023-09-29T23:25:45.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:25:45.506 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.507 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.507 DEBUG:boto:Params: {} 2023-09-29T23:25:45.507 DEBUG:boto:Token: None 2023-09-29T23:25:45.507 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/a 2023-09-29T23:25:45.508 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a3CttVooxs+Bqx5an7ZPPTIwc0s= 2023-09-29T23:25:45.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a3CttVooxs+Bqx5an7ZPPTIwc0s='} 2023-09-29T23:25:45.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2d687e73992282c-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.517 DEBUG:boto:path=/dmuhsu-66/ 2023-09-29T23:25:45.517 DEBUG:boto:auth_path=/dmuhsu-66/ 2023-09-29T23:25:45.517 DEBUG:boto:Method: HEAD 2023-09-29T23:25:45.517 DEBUG:boto:Path: /dmuhsu-66/ 2023-09-29T23:25:45.517 DEBUG:boto:Data: 2023-09-29T23:25:45.517 DEBUG:boto:Headers: {} 2023-09-29T23:25:45.518 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.518 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.518 DEBUG:boto:Params: {} 2023-09-29T23:25:45.518 DEBUG:boto:Token: None 2023-09-29T23:25:45.518 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/ 2023-09-29T23:25:45.518 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-29T23:25:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-29T23:25:45.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e5f0e39ebc22b91-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.521 DEBUG:boto:path=/dmuhsu-66/b 2023-09-29T23:25:45.521 DEBUG:boto:auth_path=/dmuhsu-66/b 2023-09-29T23:25:45.521 DEBUG:boto:Method: PUT 2023-09-29T23:25:45.521 DEBUG:boto:Path: /dmuhsu-66/b 2023-09-29T23:25:45.521 DEBUG:boto:Data: 2023-09-29T23:25:45.521 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:25:45.521 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.521 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.522 DEBUG:boto:Params: {} 2023-09-29T23:25:45.522 DEBUG:boto:Token: None 2023-09-29T23:25:45.522 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/b 2023-09-29T23:25:45.522 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mJNYFCj0QBi2XPmMv+S/VNFuOwA= 2023-09-29T23:25:45.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mJNYFCj0QBi2XPmMv+S/VNFuOwA='} 2023-09-29T23:25:45.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3747643583233cf-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.529 DEBUG:boto:path=/dmuhsu-66/ 2023-09-29T23:25:45.529 DEBUG:boto:auth_path=/dmuhsu-66/ 2023-09-29T23:25:45.529 DEBUG:boto:Method: HEAD 2023-09-29T23:25:45.529 DEBUG:boto:Path: /dmuhsu-66/ 2023-09-29T23:25:45.529 DEBUG:boto:Data: 2023-09-29T23:25:45.529 DEBUG:boto:Headers: {} 2023-09-29T23:25:45.529 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.529 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.529 DEBUG:boto:Params: {} 2023-09-29T23:25:45.530 DEBUG:boto:Token: None 2023-09-29T23:25:45.530 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/ 2023-09-29T23:25:45.530 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-29T23:25:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-29T23:25:45.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ced8b3d82eaee918-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.532 DEBUG:boto:path=/dmuhsu-66/c 2023-09-29T23:25:45.532 DEBUG:boto:auth_path=/dmuhsu-66/c 2023-09-29T23:25:45.532 DEBUG:boto:Method: PUT 2023-09-29T23:25:45.532 DEBUG:boto:Path: /dmuhsu-66/c 2023-09-29T23:25:45.532 DEBUG:boto:Data: 2023-09-29T23:25:45.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:25:45.533 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.533 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.533 DEBUG:boto:Params: {} 2023-09-29T23:25:45.533 DEBUG:boto:Token: None 2023-09-29T23:25:45.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/c 2023-09-29T23:25:45.533 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9abb6l1e9ozB++GIdXBDYQvkeLM= 2023-09-29T23:25:45.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9abb6l1e9ozB++GIdXBDYQvkeLM='} 2023-09-29T23:25:45.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000122d23955a421e02-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.579 DEBUG:boto:path=/dmuhsu-66/ 2023-09-29T23:25:45.579 DEBUG:boto:auth_path=/dmuhsu-66/ 2023-09-29T23:25:45.579 DEBUG:boto:Method: HEAD 2023-09-29T23:25:45.579 DEBUG:boto:Path: /dmuhsu-66/ 2023-09-29T23:25:45.579 DEBUG:boto:Data: 2023-09-29T23:25:45.579 DEBUG:boto:Headers: {} 2023-09-29T23:25:45.579 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.579 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.579 DEBUG:boto:Params: {} 2023-09-29T23:25:45.580 DEBUG:boto:Token: None 2023-09-29T23:25:45.580 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/ 2023-09-29T23:25:45.580 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-29T23:25:45.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-29T23:25:45.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000627e8a37b95007d0-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.582 DEBUG:boto:path=/dmuhsu-66/d 2023-09-29T23:25:45.583 DEBUG:boto:auth_path=/dmuhsu-66/d 2023-09-29T23:25:45.583 DEBUG:boto:Method: PUT 2023-09-29T23:25:45.583 DEBUG:boto:Path: /dmuhsu-66/d 2023-09-29T23:25:45.583 DEBUG:boto:Data: 2023-09-29T23:25:45.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:25:45.583 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:25:45.583 DEBUG:boto:Port: 8000 2023-09-29T23:25:45.583 DEBUG:boto:Params: {} 2023-09-29T23:25:45.583 DEBUG:boto:Token: None 2023-09-29T23:25:45.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:25:45 GMT /dmuhsu-66/d 2023-09-29T23:25:45.584 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xOAGigup0MI/AdPi1llhrsIZ74k= 2023-09-29T23:25:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xOAGigup0MI/AdPi1llhrsIZ74k='} 2023-09-29T23:25:45.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb2de5e677ed2b7f-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:25:45.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.590 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:45.636 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:45.637 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:45.688 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:25:45.704 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029945.497737_530.1", 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:25:45.497737Z" 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.705 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:25:45.706 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:25:45.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:25:45.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.707 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:45.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:45.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:45.794 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:25:45.793+0000 7efcdd615880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:45.794 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:25:45.794+0000 7efcdd615880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:45.809 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:25:45.809 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:25:45.809 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:25:45.810 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:25:45.812 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696029945.497737_530.1", 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:25:45.813 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:25:45.814 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.815 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:25:45.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:25:45.816 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:25:45.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:25:45.816 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:25:45Z" 2023-09-29T23:25:45.816 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:25:45.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029945.497737_530.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:25:45.497737Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:45Z" } 2023-09-29T23:25:45.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029945.497737_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:25:45.497737Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:25:45.817 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:25:45.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:25:45.817 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:25:45.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:25:45.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:45.818 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:45.863 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:45.863 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:45.913 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:45.913+0000 7f197eda2880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:45.914 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:45.914+0000 7f197eda2880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:45.936 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.937 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:25:45.938 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:25:45.939 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:25:45.940 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696029945.497737_530.1", 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:25:45.942 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:25:45.943 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:25:45.944 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:25:45.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:25:45.944 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:25:45Z" 2023-09-29T23:25:45.944 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:25:45.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696029945.497737_530.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:25:45.497737Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:45Z" } 2023-09-29T23:25:45.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029945.497737_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:25:45.497737Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:25:45.945 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:25:45.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:25:45.946 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:25:45.946 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-66 2023-09-29T23:25:45.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:25:45.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:25:45.949 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:25:46.005 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:46.005 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:46.036 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-66 2023-09-29T23:25:46.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-66 2023-09-29T23:25:46.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:25:46.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:25:46.050 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:25:46.102 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:46.102 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:46.155 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:46.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:25:51.156 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:51.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:25:56.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:25:56.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:01.157 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:01.157+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:06.158 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:06.157+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:11.159 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:11.158+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:16.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:16.160+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:21.160 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:21.160+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:26.161 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:26.161+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-29T23:26:31.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:31.163+0000 7f5f7992d880 1 bucket sync caught up with source: 2023-09-29T23:26:31.163 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.26.6, , , , 00000000001.51.6, , , 00000000001.27.6, , 00000000001.137.6] 2023-09-29T23:26:31.163 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.26.6, , , , 00000000001.51.6, , , 00000000001.27.6, , 00000000001.137.6] 2023-09-29T23:26:31.163 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:31.163+0000 7f5f7992d880 0 bucket checkpoint complete 2023-09-29T23:26:31.190 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-66 2023-09-29T23:26:31.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:26:31.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:26:31.191 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:26:31.247 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:31.247 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:31.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:31.298+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:26:36.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:36.298+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:26:41.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:41.299+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:26:46.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:46.299+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:26:51.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:51.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:26:56.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:26:56.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:01.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:01.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:06.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:06.301+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:11.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:11.302+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:16.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:16.303+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:21.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:21.304+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:26.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:26.304+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:31.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:31.305+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:36.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:36.305+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:41.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:41.306+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:46.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:46.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:51.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:51.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:27:56.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:27:56.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:01.308 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:01.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:06.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:06.308+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:11.309 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:11.309+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:16.310 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:16.310+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:21.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:21.311+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:26.311 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:26.311+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:31.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:31.312+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:36.312 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:36.312+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:41.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:41.313+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:46.313 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:46.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:51.314 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:51.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:28:56.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:28:56.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:01.315 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:01.315+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:06.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:06.315+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:11.316 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:11.317+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:16.317 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:16.317+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:21.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:21.318+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:26.318 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:26.318+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:31.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:31.319+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:36.319 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:36.319+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:41.320 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:41.320+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:46.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:46.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:51.321 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:51.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:29:56.322 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:29:56.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:01.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:01.322+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:06.323 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:06.322+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:11.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:11.324+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:16.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:16.324+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:21.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:21.325+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:26.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:26.326+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:31.326 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:31.326+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:36.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:36.327+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:41.327 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:41.327+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:46.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:46.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:51.328 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:51.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:30:56.329 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:30:56.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:01.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:01.329+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:06.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:06.330+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:11.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:11.331+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:16.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:16.331+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:21.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:21.332+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-29T23:31:26.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:26.332+0000 7f971d511880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:31:26.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:26.332+0000 7f971d511880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:31:26.333 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:26.333+0000 7f971d511880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:31:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:31:26.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-09-29T23:31:26.352 DEBUG:boto:path=/dmuhsu-67/ 2023-09-29T23:31:26.352 DEBUG:boto:auth_path=/dmuhsu-67/ 2023-09-29T23:31:26.352 DEBUG:boto:Method: PUT 2023-09-29T23:31:26.353 DEBUG:boto:Path: /dmuhsu-67/ 2023-09-29T23:31:26.353 DEBUG:boto:Data: 2023-09-29T23:31:26.353 DEBUG:boto:Headers: {} 2023-09-29T23:31:26.353 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:26.353 DEBUG:boto:Port: 8000 2023-09-29T23:31:26.353 DEBUG:boto:Params: {} 2023-09-29T23:31:26.353 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:31:26.354 DEBUG:boto:Token: None 2023-09-29T23:31:26.354 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:31:26 GMT /dmuhsu-67/ 2023-09-29T23:31:26.354 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2+cFvOnqzNH5S0Ko//aJGpyHO9Q= 2023-09-29T23:31:26.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2+cFvOnqzNH5S0Ko//aJGpyHO9Q=', 'Content-Length': '0'} 2023-09-29T23:31:26.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000094f8adcbd0d40551-0065175e4e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:26.376 DEBUG:rgw_multi.tests:created bucket=dmuhsu-67 2023-09-29T23:31:26.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.377 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:26.426 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:26.426 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:26.473 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:31:26.486 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.487 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030286.374800_534.1", 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:31:26.374800Z" 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.488 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:31:26.489 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:26.489 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:31:26.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.490 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:26.543 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:26.544 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:26.579 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:31:26.579+0000 7f3629617880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:26.580 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:31:26.580+0000 7f3629617880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:26.594 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:31:26.594 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:31:26.594 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:31:26.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:31:26.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696030286.374800_534.1", 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:31:26.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:31:26.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:31:26.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:31:26.601 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:31:26.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:31:26.601 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:31:26Z" 2023-09-29T23:31:26.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:31:26.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030286.374800_534.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:26.374800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:26Z" } 2023-09-29T23:31:26.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.374800_534.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.374800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:31:26.602 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:26.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:26.602 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:31:26.603 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:31:26.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:26.603 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:26.650 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:26.650 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:26.689 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:26.689+0000 7fe93f91e880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:26.690 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:26.690+0000 7fe93f91e880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:26.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:31:26.707 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:26.708 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030286.367071_531.1", 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:26.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:31:26.367071Z", 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:31:26.710 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:31:26.711 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:31:26.712 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:31:26.712 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:26.712 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:31:26Z" 2023-09-29T23:31:26.712 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:31:26.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030286.367071_531.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:26.367071Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:26Z" } 2023-09-29T23:31:26.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.367071_531.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.367071Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:31:26.713 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:26.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.367071_531.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:26.713 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696030286.374800_534.1 target=1_1696030286.367071_531.1 2023-09-29T23:31:31.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:31.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:31.715 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:31.767 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:31.767 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:31.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:31.805+0000 7f8238244880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:31.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:31.806+0000 7f8238244880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:31.826 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:31:31.826 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:31:31.826 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:31.827 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:31.828 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.829 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030286.374800_534.1", 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:31.830 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:31:31.831 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:31:31.832 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:31:31Z" 2023-09-29T23:31:31.833 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:31:31.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030286.374800_534.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:26.374800Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:31Z" } 2023-09-29T23:31:31.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.374800_534.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.374800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:31:31.834 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:31.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:31:31.834 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:31:31.834 DEBUG:boto:path=/dmuhsu-67/ 2023-09-29T23:31:31.834 DEBUG:boto:auth_path=/dmuhsu-67/ 2023-09-29T23:31:31.835 DEBUG:boto:Method: HEAD 2023-09-29T23:31:31.835 DEBUG:boto:Path: /dmuhsu-67/ 2023-09-29T23:31:31.835 DEBUG:boto:Data: 2023-09-29T23:31:31.835 DEBUG:boto:Headers: {} 2023-09-29T23:31:31.835 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.835 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.835 DEBUG:boto:Params: {} 2023-09-29T23:31:31.835 DEBUG:boto:Token: None 2023-09-29T23:31:31.835 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/ 2023-09-29T23:31:31.836 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-29T23:31:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-29T23:31:31.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c491cbe66d7efc99-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.838 DEBUG:boto:path=/dmuhsu-67/a 2023-09-29T23:31:31.839 DEBUG:boto:auth_path=/dmuhsu-67/a 2023-09-29T23:31:31.839 DEBUG:boto:Method: PUT 2023-09-29T23:31:31.839 DEBUG:boto:Path: /dmuhsu-67/a 2023-09-29T23:31:31.839 DEBUG:boto:Data: 2023-09-29T23:31:31.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:31:31.839 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.839 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.839 DEBUG:boto:Params: {} 2023-09-29T23:31:31.839 DEBUG:boto:Token: None 2023-09-29T23:31:31.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/a 2023-09-29T23:31:31.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8q5FYG2fbz8ivBNN7FYARQ0JwbU= 2023-09-29T23:31:31.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8q5FYG2fbz8ivBNN7FYARQ0JwbU='} 2023-09-29T23:31:31.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fab3562cbcf3c34c-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.847 DEBUG:boto:path=/dmuhsu-67/ 2023-09-29T23:31:31.847 DEBUG:boto:auth_path=/dmuhsu-67/ 2023-09-29T23:31:31.847 DEBUG:boto:Method: HEAD 2023-09-29T23:31:31.847 DEBUG:boto:Path: /dmuhsu-67/ 2023-09-29T23:31:31.847 DEBUG:boto:Data: 2023-09-29T23:31:31.847 DEBUG:boto:Headers: {} 2023-09-29T23:31:31.847 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.847 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.848 DEBUG:boto:Params: {} 2023-09-29T23:31:31.848 DEBUG:boto:Token: None 2023-09-29T23:31:31.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/ 2023-09-29T23:31:31.848 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-29T23:31:31.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-29T23:31:31.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002162e53fc1e66865-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.850 DEBUG:boto:path=/dmuhsu-67/b 2023-09-29T23:31:31.850 DEBUG:boto:auth_path=/dmuhsu-67/b 2023-09-29T23:31:31.850 DEBUG:boto:Method: PUT 2023-09-29T23:31:31.851 DEBUG:boto:Path: /dmuhsu-67/b 2023-09-29T23:31:31.851 DEBUG:boto:Data: 2023-09-29T23:31:31.851 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:31:31.851 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.851 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.851 DEBUG:boto:Params: {} 2023-09-29T23:31:31.851 DEBUG:boto:Token: None 2023-09-29T23:31:31.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/b 2023-09-29T23:31:31.851 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DZeewFRffr6GORt3/J/Tf7jhvSI= 2023-09-29T23:31:31.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DZeewFRffr6GORt3/J/Tf7jhvSI='} 2023-09-29T23:31:31.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e847ba09d918dae-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.861 DEBUG:boto:path=/dmuhsu-67/ 2023-09-29T23:31:31.861 DEBUG:boto:auth_path=/dmuhsu-67/ 2023-09-29T23:31:31.861 DEBUG:boto:Method: HEAD 2023-09-29T23:31:31.861 DEBUG:boto:Path: /dmuhsu-67/ 2023-09-29T23:31:31.861 DEBUG:boto:Data: 2023-09-29T23:31:31.861 DEBUG:boto:Headers: {} 2023-09-29T23:31:31.861 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.861 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.861 DEBUG:boto:Params: {} 2023-09-29T23:31:31.862 DEBUG:boto:Token: None 2023-09-29T23:31:31.862 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/ 2023-09-29T23:31:31.862 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-29T23:31:31.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-29T23:31:31.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5508aa7203e5071-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.864 DEBUG:boto:path=/dmuhsu-67/c 2023-09-29T23:31:31.865 DEBUG:boto:auth_path=/dmuhsu-67/c 2023-09-29T23:31:31.865 DEBUG:boto:Method: PUT 2023-09-29T23:31:31.865 DEBUG:boto:Path: /dmuhsu-67/c 2023-09-29T23:31:31.865 DEBUG:boto:Data: 2023-09-29T23:31:31.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:31:31.865 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.865 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.865 DEBUG:boto:Params: {} 2023-09-29T23:31:31.865 DEBUG:boto:Token: None 2023-09-29T23:31:31.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/c 2023-09-29T23:31:31.866 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W9auZ3lLrjb877opk6Er/NWMMXY= 2023-09-29T23:31:31.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W9auZ3lLrjb877opk6Er/NWMMXY='} 2023-09-29T23:31:31.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005829d57de4589173-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.871 DEBUG:boto:path=/dmuhsu-67/ 2023-09-29T23:31:31.871 DEBUG:boto:auth_path=/dmuhsu-67/ 2023-09-29T23:31:31.871 DEBUG:boto:Method: HEAD 2023-09-29T23:31:31.871 DEBUG:boto:Path: /dmuhsu-67/ 2023-09-29T23:31:31.871 DEBUG:boto:Data: 2023-09-29T23:31:31.871 DEBUG:boto:Headers: {} 2023-09-29T23:31:31.871 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.871 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.871 DEBUG:boto:Params: {} 2023-09-29T23:31:31.872 DEBUG:boto:Token: None 2023-09-29T23:31:31.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/ 2023-09-29T23:31:31.872 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-29T23:31:31.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-29T23:31:31.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e40a426f2b01b03-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.874 DEBUG:boto:path=/dmuhsu-67/d 2023-09-29T23:31:31.874 DEBUG:boto:auth_path=/dmuhsu-67/d 2023-09-29T23:31:31.874 DEBUG:boto:Method: PUT 2023-09-29T23:31:31.874 DEBUG:boto:Path: /dmuhsu-67/d 2023-09-29T23:31:31.875 DEBUG:boto:Data: 2023-09-29T23:31:31.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:31:31.875 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:31:31.875 DEBUG:boto:Port: 8000 2023-09-29T23:31:31.875 DEBUG:boto:Params: {} 2023-09-29T23:31:31.875 DEBUG:boto:Token: None 2023-09-29T23:31:31.875 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:31:31 GMT /dmuhsu-67/d 2023-09-29T23:31:31.875 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hvuoZ4KrccTPkKJxcDnf1hnrIWk= 2023-09-29T23:31:31.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvuoZ4KrccTPkKJxcDnf1hnrIWk='} 2023-09-29T23:31:31.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5124fe1d62ed650-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:31.923 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-67 2023-09-29T23:31:31.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:31.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:31.923 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:31:31.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:31.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:32.017 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-67 2023-09-29T23:31:32.032 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-67 2023-09-29T23:31:32.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:32.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:32.033 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:31:32.084 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:32.085 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:32.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:32.129+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-29T23:31:37.129 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:37.129+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-29T23:31:42.130 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:42.130+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-29T23:31:47.131 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:47.131+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-29T23:31:52.132 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:52.131+0000 7f7032b4b880 1 bucket sync caught up with source: 2023-09-29T23:31:52.132 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.17.6, , , , 00000000001.48.6, , , 00000000001.31.6, , 00000000001.46.6] 2023-09-29T23:31:52.133 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.17.6, , , , 00000000001.48.6, , , 00000000001.31.6, , 00000000001.46.6] 2023-09-29T23:31:52.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:52.131+0000 7f7032b4b880 0 bucket checkpoint complete 2023-09-29T23:31:52.153 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-67 2023-09-29T23:31:52.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:52.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:31:52.154 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:31:52.211 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:52.211 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:52.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:52.258+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:31:57.259 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:31:57.259+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:02.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:02.259+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:07.260 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:07.260+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:12.261 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:12.261+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:17.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:17.262+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:22.262 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:22.262+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:27.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:27.263+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:32.263 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:32.263+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:37.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:37.264+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:42.264 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:42.264+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:47.265 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:47.265+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:52.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:52.265+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:32:57.266 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:32:57.266+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:02.267 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:02.266+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:07.268 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:07.268+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:12.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:12.268+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:17.269 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:17.269+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:22.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:22.270+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:27.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:27.270+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:32.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:32.271+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:37.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:37.271+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:42.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:42.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:47.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:47.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:52.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:52.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:33:57.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:33:57.273+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:02.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:02.273+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:07.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:07.274+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:12.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:12.275+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:17.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:17.275+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:22.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:22.276+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:27.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:27.277+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:32.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:32.277+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:37.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:37.278+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:42.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:42.279+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:47.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:47.278+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:52.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:52.279+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:34:57.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:34:57.280+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:02.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:02.280+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:07.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:07.281+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:12.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:12.282+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:17.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:17.283+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:22.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:22.283+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:27.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:27.284+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:32.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:32.284+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:37.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:37.285+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:42.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:42.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:47.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:47.285+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:52.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:52.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:35:57.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:35:57.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:02.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:02.288+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:07.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:07.288+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:12.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:12.289+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:17.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:17.290+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:22.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:22.290+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:27.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:27.291+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:32.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:32.292+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:37.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:37.292+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:42.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:42.293+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-29T23:36:47.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:47.292+0000 7f67a3b23880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:36:47.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:47.292+0000 7f67a3b23880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:47.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:47.292+0000 7f67a3b23880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:36:47.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2023-09-29T23:36:47.315 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:36:47.315 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:36:47.315 DEBUG:boto:Method: PUT 2023-09-29T23:36:47.315 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:36:47.315 DEBUG:boto:Data: 2023-09-29T23:36:47.315 DEBUG:boto:Headers: {} 2023-09-29T23:36:47.316 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:47.316 DEBUG:boto:Port: 8000 2023-09-29T23:36:47.316 DEBUG:boto:Params: {} 2023-09-29T23:36:47.316 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:36:47.316 DEBUG:boto:Token: None 2023-09-29T23:36:47.316 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:36:47 GMT /dmuhsu-68/ 2023-09-29T23:36:47.317 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ikUh7aGPyCAkgcUlsuHq6TPV/PE= 2023-09-29T23:36:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ikUh7aGPyCAkgcUlsuHq6TPV/PE=', 'Content-Length': '0'} 2023-09-29T23:36:47.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009be27aadf440ebbc-0065175f8f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:47.342 DEBUG:rgw_multi.tests:created bucket=dmuhsu-68 2023-09-29T23:36:47.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.344 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:47.392 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:47.392 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:47.448 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:36:47.465 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:36:47.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:36:47.465 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030607.338642_550.1", 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:36:47.338642Z" 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.467 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:36:47.467 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:36:47.467 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.467 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:36:47.468 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:36:47.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:36:47.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.468 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:47.522 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:47.522 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:47.558 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:36:47.557+0000 7f22e8b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:47.558 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:36:47.558+0000 7f22e8b36880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:36:47.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:36:47.574 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:36:47.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:36:47.576 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696030607.338642_550.1", 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:36:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:36:47.578 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:36:47Z" 2023-09-29T23:36:47.579 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:36:47.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030607.338642_550.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:36:47.338642Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:36:47Z" } 2023-09-29T23:36:47.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030607.338642_550.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:36:47.338642Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:36:47.580 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:36:47.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:36:47.581 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:36:47.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:36:47.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:47.581 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:47.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:47.626 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:47.660+0000 7f499cc30880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:47.661 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:36:47.661+0000 7f499cc30880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:47.687 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:36:47.688 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:36:47.689 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:36:47.690 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030607.338642_550.1", 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-29T23:36:47.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:36:47.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:36:47.694 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:36:47Z" 2023-09-29T23:36:47.695 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:36:47.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030607.338642_550.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:36:47.338642Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:36:47Z" } 2023-09-29T23:36:47.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030607.338642_550.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:36:47.338642Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:36:47.696 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:36:47.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:36:47.696 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:36:47.696 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-29T23:36:47.697 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:47.697 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T23:36:47.697+0000 7f6fade67640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 89189) UID: 0 2023-09-29T23:36:47.698 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T23:36:47.697+0000 7f6fb4212880 -1 shutting down 2023-09-29T23:36:48.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-29T23:36:53.798 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-29T23:36:53.798 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T23:36:53.798 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:53.930 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-29T23:36:53.799+0000 7f0f322db640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 90724) UID: 0 2023-09-29T23:36:53.930 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-29T23:36:53.799+0000 7f0f38434880 -1 shutting down 2023-09-29T23:36:53.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2023-09-29T23:36:55.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T23:36:59.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-29T23:36:59.900 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T23:36:59.901 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:36:59.901 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:36:59.901 DEBUG:boto:Method: HEAD 2023-09-29T23:36:59.901 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:36:59.901 DEBUG:boto:Data: 2023-09-29T23:36:59.901 DEBUG:boto:Headers: {} 2023-09-29T23:36:59.901 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.901 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.901 DEBUG:boto:Params: {} 2023-09-29T23:36:59.902 DEBUG:boto:Token: None 2023-09-29T23:36:59.902 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/ 2023-09-29T23:36:59.902 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-29T23:36:59.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-29T23:36:59.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3310a1464593303-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.905 DEBUG:boto:path=/dmuhsu-68/a 2023-09-29T23:36:59.905 DEBUG:boto:auth_path=/dmuhsu-68/a 2023-09-29T23:36:59.905 DEBUG:boto:Method: PUT 2023-09-29T23:36:59.906 DEBUG:boto:Path: /dmuhsu-68/a 2023-09-29T23:36:59.906 DEBUG:boto:Data: 2023-09-29T23:36:59.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:59.906 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.906 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.906 DEBUG:boto:Params: {} 2023-09-29T23:36:59.906 DEBUG:boto:Token: None 2023-09-29T23:36:59.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/a 2023-09-29T23:36:59.906 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rs7k1hOIkKUKL2oGz64j34ACdAk= 2023-09-29T23:36:59.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rs7k1hOIkKUKL2oGz64j34ACdAk='} 2023-09-29T23:36:59.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6fae89d09930162-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.917 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:36:59.917 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:36:59.917 DEBUG:boto:Method: HEAD 2023-09-29T23:36:59.918 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:36:59.918 DEBUG:boto:Data: 2023-09-29T23:36:59.918 DEBUG:boto:Headers: {} 2023-09-29T23:36:59.918 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.918 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.918 DEBUG:boto:Params: {} 2023-09-29T23:36:59.918 DEBUG:boto:Token: None 2023-09-29T23:36:59.918 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/ 2023-09-29T23:36:59.918 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-29T23:36:59.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-29T23:36:59.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060ba954bf05675c8-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.921 DEBUG:boto:path=/dmuhsu-68/b 2023-09-29T23:36:59.921 DEBUG:boto:auth_path=/dmuhsu-68/b 2023-09-29T23:36:59.921 DEBUG:boto:Method: PUT 2023-09-29T23:36:59.921 DEBUG:boto:Path: /dmuhsu-68/b 2023-09-29T23:36:59.921 DEBUG:boto:Data: 2023-09-29T23:36:59.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:59.922 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.922 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.922 DEBUG:boto:Params: {} 2023-09-29T23:36:59.922 DEBUG:boto:Token: None 2023-09-29T23:36:59.922 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/b 2023-09-29T23:36:59.922 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9Mie76D5hf86w9ZhZicSAezy/gs= 2023-09-29T23:36:59.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9Mie76D5hf86w9ZhZicSAezy/gs='} 2023-09-29T23:36:59.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a321b37be698c8b7-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.932 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:36:59.932 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:36:59.932 DEBUG:boto:Method: HEAD 2023-09-29T23:36:59.932 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:36:59.932 DEBUG:boto:Data: 2023-09-29T23:36:59.933 DEBUG:boto:Headers: {} 2023-09-29T23:36:59.933 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.933 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.933 DEBUG:boto:Params: {} 2023-09-29T23:36:59.933 DEBUG:boto:Token: None 2023-09-29T23:36:59.933 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/ 2023-09-29T23:36:59.933 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-29T23:36:59.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-29T23:36:59.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d00ec4d9b3039945-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.936 DEBUG:boto:path=/dmuhsu-68/c 2023-09-29T23:36:59.936 DEBUG:boto:auth_path=/dmuhsu-68/c 2023-09-29T23:36:59.936 DEBUG:boto:Method: PUT 2023-09-29T23:36:59.936 DEBUG:boto:Path: /dmuhsu-68/c 2023-09-29T23:36:59.936 DEBUG:boto:Data: 2023-09-29T23:36:59.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:59.937 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.937 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.937 DEBUG:boto:Params: {} 2023-09-29T23:36:59.937 DEBUG:boto:Token: None 2023-09-29T23:36:59.937 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/c 2023-09-29T23:36:59.937 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ml+cigcke75TEI/XWu772YQJx58= 2023-09-29T23:36:59.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ml+cigcke75TEI/XWu772YQJx58='} 2023-09-29T23:36:59.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c956569061445647-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.983 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:36:59.983 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:36:59.983 DEBUG:boto:Method: HEAD 2023-09-29T23:36:59.983 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:36:59.983 DEBUG:boto:Data: 2023-09-29T23:36:59.983 DEBUG:boto:Headers: {} 2023-09-29T23:36:59.984 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.984 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.984 DEBUG:boto:Params: {} 2023-09-29T23:36:59.984 DEBUG:boto:Token: None 2023-09-29T23:36:59.984 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/ 2023-09-29T23:36:59.984 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-29T23:36:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-29T23:36:59.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009263f87a790ee49c-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.987 DEBUG:boto:path=/dmuhsu-68/d 2023-09-29T23:36:59.987 DEBUG:boto:auth_path=/dmuhsu-68/d 2023-09-29T23:36:59.987 DEBUG:boto:Method: PUT 2023-09-29T23:36:59.987 DEBUG:boto:Path: /dmuhsu-68/d 2023-09-29T23:36:59.987 DEBUG:boto:Data: 2023-09-29T23:36:59.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:59.987 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:36:59.987 DEBUG:boto:Port: 8000 2023-09-29T23:36:59.987 DEBUG:boto:Params: {} 2023-09-29T23:36:59.988 DEBUG:boto:Token: None 2023-09-29T23:36:59.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:36:59 GMT /dmuhsu-68/d 2023-09-29T23:36:59.988 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ebq6iDgGrz0AJ2GQlnMtmJD9HEI= 2023-09-29T23:36:59.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ebq6iDgGrz0AJ2GQlnMtmJD9HEI='} 2023-09-29T23:36:59.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5ba6400891084f3-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:59.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dmuhsu-68', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:59.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dmuhsu-68', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:59.999 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dmuhsu-68 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:37:00.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:37:00.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:37:00.113 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-29T23:37:00.114 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-68 2023-09-29T23:37:00.115 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 4 2023-09-29T23:37:00.163 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:37:00.163 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:37:00.164 DEBUG:boto:Method: HEAD 2023-09-29T23:37:00.164 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:37:00.164 DEBUG:boto:Data: 2023-09-29T23:37:00.164 DEBUG:boto:Headers: {} 2023-09-29T23:37:00.164 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.164 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.164 DEBUG:boto:Params: {} 2023-09-29T23:37:00.164 DEBUG:boto:Token: None 2023-09-29T23:37:00.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/ 2023-09-29T23:37:00.165 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-29T23:37:00.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-29T23:37:00.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd27de35143e6b2e-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.167 DEBUG:boto:path=/dmuhsu-68/e 2023-09-29T23:37:00.167 DEBUG:boto:auth_path=/dmuhsu-68/e 2023-09-29T23:37:00.167 DEBUG:boto:Method: PUT 2023-09-29T23:37:00.167 DEBUG:boto:Path: /dmuhsu-68/e 2023-09-29T23:37:00.167 DEBUG:boto:Data: 2023-09-29T23:37:00.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:37:00.167 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.167 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.167 DEBUG:boto:Params: {} 2023-09-29T23:37:00.168 DEBUG:boto:Token: None 2023-09-29T23:37:00.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/e 2023-09-29T23:37:00.168 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HJee3Nwa98e62BEMM4sXn5aKxXo= 2023-09-29T23:37:00.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HJee3Nwa98e62BEMM4sXn5aKxXo='} 2023-09-29T23:37:00.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f979fa10883c8c4d-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.215 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:37:00.215 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:37:00.215 DEBUG:boto:Method: HEAD 2023-09-29T23:37:00.215 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:37:00.215 DEBUG:boto:Data: 2023-09-29T23:37:00.215 DEBUG:boto:Headers: {} 2023-09-29T23:37:00.215 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.215 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.216 DEBUG:boto:Params: {} 2023-09-29T23:37:00.216 DEBUG:boto:Token: None 2023-09-29T23:37:00.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/ 2023-09-29T23:37:00.216 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-29T23:37:00.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-29T23:37:00.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ff575d95843c665-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.218 DEBUG:boto:path=/dmuhsu-68/f 2023-09-29T23:37:00.219 DEBUG:boto:auth_path=/dmuhsu-68/f 2023-09-29T23:37:00.219 DEBUG:boto:Method: PUT 2023-09-29T23:37:00.219 DEBUG:boto:Path: /dmuhsu-68/f 2023-09-29T23:37:00.219 DEBUG:boto:Data: 2023-09-29T23:37:00.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:37:00.219 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.219 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.219 DEBUG:boto:Params: {} 2023-09-29T23:37:00.220 DEBUG:boto:Token: None 2023-09-29T23:37:00.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/f 2023-09-29T23:37:00.220 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ue0NjbW0XQQebi55AX0EPt3uFcY= 2023-09-29T23:37:00.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ue0NjbW0XQQebi55AX0EPt3uFcY='} 2023-09-29T23:37:00.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e13d2aa9e886caaa-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.231 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:37:00.231 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:37:00.231 DEBUG:boto:Method: HEAD 2023-09-29T23:37:00.231 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:37:00.231 DEBUG:boto:Data: 2023-09-29T23:37:00.231 DEBUG:boto:Headers: {} 2023-09-29T23:37:00.231 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.231 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.232 DEBUG:boto:Params: {} 2023-09-29T23:37:00.232 DEBUG:boto:Token: None 2023-09-29T23:37:00.232 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/ 2023-09-29T23:37:00.232 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-29T23:37:00.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-29T23:37:00.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83affcf51f89f2d-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.234 DEBUG:boto:path=/dmuhsu-68/g 2023-09-29T23:37:00.234 DEBUG:boto:auth_path=/dmuhsu-68/g 2023-09-29T23:37:00.235 DEBUG:boto:Method: PUT 2023-09-29T23:37:00.235 DEBUG:boto:Path: /dmuhsu-68/g 2023-09-29T23:37:00.235 DEBUG:boto:Data: 2023-09-29T23:37:00.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:37:00.235 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.235 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.235 DEBUG:boto:Params: {} 2023-09-29T23:37:00.235 DEBUG:boto:Token: None 2023-09-29T23:37:00.235 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/g 2023-09-29T23:37:00.236 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:O22dzXi3GGeBrWbw6UDzgYceglA= 2023-09-29T23:37:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O22dzXi3GGeBrWbw6UDzgYceglA='} 2023-09-29T23:37:00.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a80c2556caa0c810-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.243 DEBUG:boto:path=/dmuhsu-68/ 2023-09-29T23:37:00.243 DEBUG:boto:auth_path=/dmuhsu-68/ 2023-09-29T23:37:00.243 DEBUG:boto:Method: HEAD 2023-09-29T23:37:00.243 DEBUG:boto:Path: /dmuhsu-68/ 2023-09-29T23:37:00.243 DEBUG:boto:Data: 2023-09-29T23:37:00.243 DEBUG:boto:Headers: {} 2023-09-29T23:37:00.243 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.243 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.243 DEBUG:boto:Params: {} 2023-09-29T23:37:00.244 DEBUG:boto:Token: None 2023-09-29T23:37:00.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/ 2023-09-29T23:37:00.244 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-29T23:37:00.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-29T23:37:00.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113e0066b620e476-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.246 DEBUG:boto:path=/dmuhsu-68/h 2023-09-29T23:37:00.246 DEBUG:boto:auth_path=/dmuhsu-68/h 2023-09-29T23:37:00.246 DEBUG:boto:Method: PUT 2023-09-29T23:37:00.246 DEBUG:boto:Path: /dmuhsu-68/h 2023-09-29T23:37:00.246 DEBUG:boto:Data: 2023-09-29T23:37:00.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:37:00.247 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:37:00.247 DEBUG:boto:Port: 8000 2023-09-29T23:37:00.247 DEBUG:boto:Params: {} 2023-09-29T23:37:00.247 DEBUG:boto:Token: None 2023-09-29T23:37:00.247 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:37:00 GMT /dmuhsu-68/h 2023-09-29T23:37:00.247 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UaxC+54cQbV+RqY0hm5G4dRWbUc= 2023-09-29T23:37:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UaxC+54cQbV+RqY0hm5G4dRWbUc='} 2023-09-29T23:37:00.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007345730c5bf46598-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:37:00.295 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T23:37:00.295 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T23:37:00.298 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T23:37:00.298 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T23:37:00.319 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:37:00.319 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:37:00.320 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:37:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T23:37:00.322 INFO:teuthology.orchestra.run.smithi184.stderr:curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T23:37:01.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T23:37:01.323 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T23:37:01.342 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:37:01.343 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:37:01.343 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T23:37:01.345 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-29T23:37:01.346 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T23:37:01.346 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T23:37:01.348 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T23:37:01.349 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2023-09-29T23:37:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T23:37:01.392 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:37:01.393 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:37:01.393 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:37:01.393 INFO:teuthology.orchestra.run.smithi175.stderr:curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T23:37:02.394 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2023-09-29T23:37:02.413 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:37:02.413 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:37:02.413 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T23:37:02.414 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2023-09-29T23:37:02.415 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-68 2023-09-29T23:37:02.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:37:02.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:37:02.415 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:37:02.474 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:37:02.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:37:02.508 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-68 2023-09-29T23:37:02.523 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-68 2023-09-29T23:37:02.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:37:02.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:37:02.524 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:37:02.575 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:37:02.575 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:37:02.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:02.624+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:07.625 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:07.625+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:12.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:12.626+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:17.626 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:17.626+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:22.627 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:22.627+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:27.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:27.627+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:32.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:32.628+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:37.628 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:37.628+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:42.629 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:42.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:47.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:47.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:52.630 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:52.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:37:57.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:37:57.630+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:02.631 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:02.631+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:07.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:07.632+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:12.632 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:12.632+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:17.633 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:17.633+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:22.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:22.633+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:27.634 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:27.634+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:32.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:32.634+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:37.635 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:37.635+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:42.636 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:42.635+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:47.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:47.636+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:52.637 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:52.637+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:38:57.638 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:38:57.637+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:02.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:02.638+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:07.639 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:07.639+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:12.640 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:12.640+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:17.641 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:17.641+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:22.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:22.642+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:27.642 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:27.642+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:32.643 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:32.643+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:37.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:37.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:42.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:42.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:47.645 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:47.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:52.646 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:52.645+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:39:57.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:57.647+0000 7ff739e0f880 1 bucket sync caught up with source: 2023-09-29T23:39:57.648 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [00000000003.27.6, 00000000002.36.6, 00000000002.28.6] 2023-09-29T23:39:57.648 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [00000000003.27.6, 00000000002.36.6, 00000000002.28.6] 2023-09-29T23:39:57.648 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:57.647+0000 7ff739e0f880 0 bucket checkpoint complete 2023-09-29T23:39:57.676 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-68 2023-09-29T23:39:57.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:39:57.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:39:57.678 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:39:57.734 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:39:57.735 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:39:57.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:39:57.790+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:02.790 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:02.790+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:07.791 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:07.791+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:12.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:12.792+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:17.792 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:17.792+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:22.793 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:22.793+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:27.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:27.793+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:32.794 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:32.794+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:37.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:37.795+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:42.795 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:42.794+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:47.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:47.795+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:52.796 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:52.796+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:40:57.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:40:57.797+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:02.797 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:02.797+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:07.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:07.798+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:12.798 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:12.798+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:17.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:17.799+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:22.799 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:22.799+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:27.800 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:27.800+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:32.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:32.800+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:37.801 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:37.801+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:42.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:42.801+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:47.802 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:47.802+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:52.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:52.802+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:41:57.803 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:41:57.803+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:02.804 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:02.804+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:07.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:07.804+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:12.805 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:12.805+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:17.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:17.806+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:22.806 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:22.806+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:27.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:27.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:32.807 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:32.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:37.808 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:37.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:42.809 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:42.808+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:47.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:47.808+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:52.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:52.810+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:42:57.810 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:42:57.810+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:02.811 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:02.811+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:07.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:07.811+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:12.812 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:12.812+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:17.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:17.813+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:22.813 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:22.813+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:27.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:27.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:32.814 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:32.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:37.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:37.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:42.815 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:42.815+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:47.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:47.815+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:52.816 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:52.816+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:43:57.817 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:43:57.817+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:02.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:02.817+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:07.818 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:07.818+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:12.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:12.818+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:17.819 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:17.819+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:22.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:22.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:27.820 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:27.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:32.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:32.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:37.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:42.822 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:42.821+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:47.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:47.822+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-29T23:44:52.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:52.823+0000 7f544ed33880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:44:52.823 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:52.823+0000 7f544ed33880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:44:52.824 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:52.823+0000 7f544ed33880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:44:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:44:52.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2023-09-29T23:44:52.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:52.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:52.844 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:52.895 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:52.895 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:52.930 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:44:52.944 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:44:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:44:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030620.120629_554.1", 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:37:00.120629Z" 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:44:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:52.946 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:44:52.947 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030620.120629_554.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:52.947 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:44:52.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:52.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:52.947 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.000 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.001 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.037 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:44:53.036+0000 7fa0f4929880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.037 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:44:53.037+0000 7fa0f4929880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:44:53.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:44:53.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.057 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696030620.120629_554.1", 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:37:00.120629Z", 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:44:53.059 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:44:53Z" 2023-09-29T23:44:53.060 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:44:53.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030620.120629_554.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:37:00.120629Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:44:53Z" } 2023-09-29T23:44:53.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030620.120629_554.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:37:00.120629Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:44:53.062 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030620.120629_554.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030620.120629_554.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.062 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:44:53.062 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:44:53.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.063 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.109 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:53.148+0000 7f7103929880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.148 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:53.148+0000 7f7103929880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:44:53.169 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:44:53.170 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.171 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:44:53.172 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696030620.120629_554.1", 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:37:00.120629Z", 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:44:53.174 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:44:53Z" 2023-09-29T23:44:53.175 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:44:53.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696030620.120629_554.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:37:00.120629Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:44:53Z" } 2023-09-29T23:44:53.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030620.120629_554.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:37:00.120629Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:44:53.177 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030620.120629_554.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030620.120629_554.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:44:53.177 DEBUG:boto:path=/ 2023-09-29T23:44:53.177 DEBUG:boto:auth_path=/ 2023-09-29T23:44:53.177 DEBUG:boto:Method: GET 2023-09-29T23:44:53.177 DEBUG:boto:Path: / 2023-09-29T23:44:53.178 DEBUG:boto:Data: 2023-09-29T23:44:53.178 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.178 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.178 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.178 DEBUG:boto:Params: {} 2023-09-29T23:44:53.178 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:44:53.179 DEBUG:boto:Token: None 2023-09-29T23:44:53.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:44:53 GMT / 2023-09-29T23:44:53.179 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8= 2023-09-29T23:44:53.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8=', 'Content-Length': '0'} 2023-09-29T23:44:53.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da710123f84330f5-0065176175-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.186 DEBUG:boto:path=/ 2023-09-29T23:44:53.186 DEBUG:boto:auth_path=/ 2023-09-29T23:44:53.186 DEBUG:boto:Method: GET 2023-09-29T23:44:53.186 DEBUG:boto:Path: / 2023-09-29T23:44:53.186 DEBUG:boto:Data: 2023-09-29T23:44:53.186 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.186 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-29T23:44:53.186 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.187 DEBUG:boto:Params: {} 2023-09-29T23:44:53.187 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:44:53.187 DEBUG:boto:Token: None 2023-09-29T23:44:53.187 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:44:53 GMT / 2023-09-29T23:44:53.187 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8= 2023-09-29T23:44:53.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8=', 'Content-Length': '0'} 2023-09-29T23:44:53.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec0a5b9187f4c52-0065176175-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.195 DEBUG:boto:path=/ 2023-09-29T23:44:53.195 DEBUG:boto:auth_path=/ 2023-09-29T23:44:53.195 DEBUG:boto:Method: GET 2023-09-29T23:44:53.195 DEBUG:boto:Path: / 2023-09-29T23:44:53.195 DEBUG:boto:Data: 2023-09-29T23:44:53.195 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.195 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-29T23:44:53.195 DEBUG:boto:Port: 8001 2023-09-29T23:44:53.195 DEBUG:boto:Params: {} 2023-09-29T23:44:53.196 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:44:53.196 DEBUG:boto:Token: None 2023-09-29T23:44:53.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:44:53 GMT / 2023-09-29T23:44:53.196 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8= 2023-09-29T23:44:53.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sich+6Qwhd63AZxtAABJ+QDLzy8=', 'Content-Length': '0'} 2023-09-29T23:44:53.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca6f9d9fa1f8886-0065176175-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-09-29T23:44:53.204 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.205 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.205 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.205 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.205 DEBUG:boto:Data: 2023-09-29T23:44:53.205 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.205 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.205 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.205 DEBUG:boto:Params: {} 2023-09-29T23:44:53.205 DEBUG:boto:Token: None 2023-09-29T23:44:53.206 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.206 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+B6HiDMlAfTJpJt9KANVvpYAmBw= 2023-09-29T23:44:53.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+B6HiDMlAfTJpJt9KANVvpYAmBw=', 'Content-Length': '0'} 2023-09-29T23:44:53.226 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000bf36c8c04e95260-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.226 DEBUG:rgw_multi.tests:created bucket=dmuhsu-69 2023-09-29T23:44:53.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.227 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.272 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.272 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.319 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:44:53.338 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:44:53.339 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031093.225022_558.1", 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:44:53.225022Z" 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.340 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:44:53.341 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.342 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:44:53.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.342 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.392 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.392 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.422 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:44:53.421+0000 7f4f5d067880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.422 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:44:53.422+0000 7f4f5d067880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.437 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:44:53.437 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:44:53.437 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:44:53.437 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031093.225022_558.1", 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:44:53.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:44:53Z" 2023-09-29T23:44:53.443 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:44:53.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031093.225022_558.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:44:53.225022Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:44:53Z" } 2023-09-29T23:44:53.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031093.225022_558.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:44:53.225022Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:44:53.446 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:44:53.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:44:53.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.447 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.492 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.522 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:53.522+0000 7fb56cb32880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.523 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:53.523+0000 7fb56cb32880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:44:53.537 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:44:53.537 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:44:53.537 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:44:53.537 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:44:53.537 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:44:53.538 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.539 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.540 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031093.225022_558.1", 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.541 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:44:53.542 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:44:53Z" 2023-09-29T23:44:53.543 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:44:53.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031093.225022_558.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:44:53.225022Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:44:53Z" } 2023-09-29T23:44:53.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031093.225022_558.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:44:53.225022Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:44:53.545 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:44:53.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:44:53.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:44:53.545 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:44:53.593 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:53.593 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:53.650 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:44:53.650 INFO:teuthology.orchestra.run.smithi175.stdout: "objs": [ 2023-09-29T23:44:53.650 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000001", 2023-09-29T23:44:53.650 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000005", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000012", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000004", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000011", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000000", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000010", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000003", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000013", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000002", 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: "obj00000000000000000006" 2023-09-29T23:44:53.651 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:44:53.652 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:44:53.652 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.653 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.653 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.653 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.653 DEBUG:boto:Data: 2023-09-29T23:44:53.653 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.653 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.653 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.653 DEBUG:boto:Params: {} 2023-09-29T23:44:53.654 DEBUG:boto:Token: None 2023-09-29T23:44:53.654 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.654 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b21ce563c5bf3ecb-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.656 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000002 2023-09-29T23:44:53.656 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000002 2023-09-29T23:44:53.657 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.657 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000002 2023-09-29T23:44:53.657 DEBUG:boto:Data: 2023-09-29T23:44:53.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.657 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.657 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.657 DEBUG:boto:Params: {} 2023-09-29T23:44:53.657 DEBUG:boto:Token: None 2023-09-29T23:44:53.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000002 2023-09-29T23:44:53.658 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:d/NSJ1QX1VsJkX7P6SSbux+SHP0= 2023-09-29T23:44:53.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d/NSJ1QX1VsJkX7P6SSbux+SHP0='} 2023-09-29T23:44:53.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049015d764d39b796-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.666 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.666 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.666 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.666 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.666 DEBUG:boto:Data: 2023-09-29T23:44:53.666 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.666 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.666 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.667 DEBUG:boto:Params: {} 2023-09-29T23:44:53.667 DEBUG:boto:Token: None 2023-09-29T23:44:53.667 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.667 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac0d6e659b629782-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.669 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000001 2023-09-29T23:44:53.669 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000001 2023-09-29T23:44:53.669 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.670 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000001 2023-09-29T23:44:53.670 DEBUG:boto:Data: 2023-09-29T23:44:53.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.670 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.670 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.670 DEBUG:boto:Params: {} 2023-09-29T23:44:53.670 DEBUG:boto:Token: None 2023-09-29T23:44:53.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000001 2023-09-29T23:44:53.670 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GwxN7OJxohZbXwdAmdv7zAVvZSg= 2023-09-29T23:44:53.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GwxN7OJxohZbXwdAmdv7zAVvZSg='} 2023-09-29T23:44:53.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005668f7923108e714-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.679 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.679 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.679 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.679 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.679 DEBUG:boto:Data: 2023-09-29T23:44:53.679 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.680 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.680 DEBUG:boto:Params: {} 2023-09-29T23:44:53.680 DEBUG:boto:Token: None 2023-09-29T23:44:53.680 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000610c0a3fb6bb20a4-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.682 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000003 2023-09-29T23:44:53.682 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000003 2023-09-29T23:44:53.683 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.683 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000003 2023-09-29T23:44:53.683 DEBUG:boto:Data: 2023-09-29T23:44:53.683 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.683 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.683 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.683 DEBUG:boto:Params: {} 2023-09-29T23:44:53.683 DEBUG:boto:Token: None 2023-09-29T23:44:53.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000003 2023-09-29T23:44:53.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3t0Acpx1qOgBbMo5CUk9SvaDPaU= 2023-09-29T23:44:53.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3t0Acpx1qOgBbMo5CUk9SvaDPaU='} 2023-09-29T23:44:53.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090353d9c6ded4433-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.727 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.727 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.727 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.727 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.727 DEBUG:boto:Data: 2023-09-29T23:44:53.727 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.727 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.728 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.728 DEBUG:boto:Params: {} 2023-09-29T23:44:53.728 DEBUG:boto:Token: None 2023-09-29T23:44:53.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000619209e4ef4875df-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.730 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000004 2023-09-29T23:44:53.731 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000004 2023-09-29T23:44:53.731 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.731 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000004 2023-09-29T23:44:53.731 DEBUG:boto:Data: 2023-09-29T23:44:53.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.731 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.731 DEBUG:boto:Params: {} 2023-09-29T23:44:53.732 DEBUG:boto:Token: None 2023-09-29T23:44:53.732 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000004 2023-09-29T23:44:53.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kQRvf9BFur9vbZC6DWyxRT6uA8g= 2023-09-29T23:44:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQRvf9BFur9vbZC6DWyxRT6uA8g='} 2023-09-29T23:44:53.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ab1984f584774fb-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.742 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.743 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.743 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.743 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.743 DEBUG:boto:Data: 2023-09-29T23:44:53.743 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.743 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.743 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.743 DEBUG:boto:Params: {} 2023-09-29T23:44:53.743 DEBUG:boto:Token: None 2023-09-29T23:44:53.744 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.744 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090b3bc1e4d3a5c49-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.746 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000006 2023-09-29T23:44:53.747 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000006 2023-09-29T23:44:53.747 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.747 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000006 2023-09-29T23:44:53.747 DEBUG:boto:Data: 2023-09-29T23:44:53.747 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.747 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.747 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.747 DEBUG:boto:Params: {} 2023-09-29T23:44:53.748 DEBUG:boto:Token: None 2023-09-29T23:44:53.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000006 2023-09-29T23:44:53.748 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zE4YVeYneOYbWCFs5rkDNPTGpnE= 2023-09-29T23:44:53.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zE4YVeYneOYbWCFs5rkDNPTGpnE='} 2023-09-29T23:44:53.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a560917238603f5-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.757 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.757 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.758 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.758 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.758 DEBUG:boto:Data: 2023-09-29T23:44:53.758 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.759 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.759 DEBUG:boto:Params: {} 2023-09-29T23:44:53.759 DEBUG:boto:Token: None 2023-09-29T23:44:53.760 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000175d9877261392a9-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.763 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000011 2023-09-29T23:44:53.764 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000011 2023-09-29T23:44:53.764 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.764 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000011 2023-09-29T23:44:53.764 DEBUG:boto:Data: 2023-09-29T23:44:53.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.765 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.765 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.766 DEBUG:boto:Params: {} 2023-09-29T23:44:53.766 DEBUG:boto:Token: None 2023-09-29T23:44:53.767 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000011 2023-09-29T23:44:53.767 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OIPxxFwM1bBPsI7vYkMHU4d3GGw= 2023-09-29T23:44:53.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OIPxxFwM1bBPsI7vYkMHU4d3GGw='} 2023-09-29T23:44:53.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075095bb29ea29963-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.811 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.811 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.811 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.812 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.812 DEBUG:boto:Data: 2023-09-29T23:44:53.812 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.812 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.812 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.813 DEBUG:boto:Params: {} 2023-09-29T23:44:53.813 DEBUG:boto:Token: None 2023-09-29T23:44:53.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.814 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058c69456c750c7ce-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.817 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000010 2023-09-29T23:44:53.818 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000010 2023-09-29T23:44:53.818 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.818 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000010 2023-09-29T23:44:53.818 DEBUG:boto:Data: 2023-09-29T23:44:53.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.819 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.819 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.819 DEBUG:boto:Params: {} 2023-09-29T23:44:53.820 DEBUG:boto:Token: None 2023-09-29T23:44:53.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000010 2023-09-29T23:44:53.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:aFR71gJbLKxfhTgvJ1iTO1D863E= 2023-09-29T23:44:53.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFR71gJbLKxfhTgvJ1iTO1D863E='} 2023-09-29T23:44:53.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000152f4ce9d4085105-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.829 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.829 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.829 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.830 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.830 DEBUG:boto:Data: 2023-09-29T23:44:53.830 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.830 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.831 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.831 DEBUG:boto:Params: {} 2023-09-29T23:44:53.831 DEBUG:boto:Token: None 2023-09-29T23:44:53.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.832 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cad785acebd236f-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.835 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000000 2023-09-29T23:44:53.835 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000000 2023-09-29T23:44:53.835 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.836 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000000 2023-09-29T23:44:53.836 DEBUG:boto:Data: 2023-09-29T23:44:53.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.836 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.837 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.837 DEBUG:boto:Params: {} 2023-09-29T23:44:53.837 DEBUG:boto:Token: None 2023-09-29T23:44:53.837 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000000 2023-09-29T23:44:53.838 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jmsxus/u/8/c26sUYrcWKGWbvnQ= 2023-09-29T23:44:53.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jmsxus/u/8/c26sUYrcWKGWbvnQ='} 2023-09-29T23:44:53.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c818581a4fd04b-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.887 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.887 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.887 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.888 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.888 DEBUG:boto:Data: 2023-09-29T23:44:53.888 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.888 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.888 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.888 DEBUG:boto:Params: {} 2023-09-29T23:44:53.889 DEBUG:boto:Token: None 2023-09-29T23:44:53.889 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.889 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000705eb733f3224c89-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.892 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000013 2023-09-29T23:44:53.892 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000013 2023-09-29T23:44:53.893 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.893 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000013 2023-09-29T23:44:53.893 DEBUG:boto:Data: 2023-09-29T23:44:53.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.893 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.894 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.894 DEBUG:boto:Params: {} 2023-09-29T23:44:53.894 DEBUG:boto:Token: None 2023-09-29T23:44:53.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000013 2023-09-29T23:44:53.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:J1dQvo5Tfgt8Dqwyw0siXIjZWMY= 2023-09-29T23:44:53.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J1dQvo5Tfgt8Dqwyw0siXIjZWMY='} 2023-09-29T23:44:53.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c4eac81b10524bc-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.901 DEBUG:boto:path=/dmuhsu-69/ 2023-09-29T23:44:53.901 DEBUG:boto:auth_path=/dmuhsu-69/ 2023-09-29T23:44:53.901 DEBUG:boto:Method: HEAD 2023-09-29T23:44:53.901 DEBUG:boto:Path: /dmuhsu-69/ 2023-09-29T23:44:53.901 DEBUG:boto:Data: 2023-09-29T23:44:53.902 DEBUG:boto:Headers: {} 2023-09-29T23:44:53.902 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.902 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.902 DEBUG:boto:Params: {} 2023-09-29T23:44:53.902 DEBUG:boto:Token: None 2023-09-29T23:44:53.902 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/ 2023-09-29T23:44:53.902 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-29T23:44:53.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-29T23:44:53.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8ce1f6e05adbe42-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.905 DEBUG:boto:path=/dmuhsu-69/obj00000000000000000012 2023-09-29T23:44:53.905 DEBUG:boto:auth_path=/dmuhsu-69/obj00000000000000000012 2023-09-29T23:44:53.905 DEBUG:boto:Method: PUT 2023-09-29T23:44:53.905 DEBUG:boto:Path: /dmuhsu-69/obj00000000000000000012 2023-09-29T23:44:53.905 DEBUG:boto:Data: 2023-09-29T23:44:53.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:44:53.905 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:44:53.905 DEBUG:boto:Port: 8000 2023-09-29T23:44:53.905 DEBUG:boto:Params: {} 2023-09-29T23:44:53.906 DEBUG:boto:Token: None 2023-09-29T23:44:53.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:44:53 GMT /dmuhsu-69/obj00000000000000000012 2023-09-29T23:44:53.906 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LdHjAi1Aa22FyA4XUIjIvySDDs4= 2023-09-29T23:44:53.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LdHjAi1Aa22FyA4XUIjIvySDDs4='} 2023-09-29T23:44:53.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006004e5c1c2fe192a-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:44:53.951 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-69 2023-09-29T23:44:53.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:44:53.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:44:53.951 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:44:54.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:54.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:54.035 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-69 2023-09-29T23:44:54.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-69 2023-09-29T23:44:54.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:44:54.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:44:54.050 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:44:54.102 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:44:54.102 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:44:54.140 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:54.139+0000 7f33d5729880 1 waiting to reach incremental sync.. 2023-09-29T23:44:59.141 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:44:59.140+0000 7f33d5729880 1 waiting to reach incremental sync.. 2023-09-29T23:45:04.142 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:04.142+0000 7f33d5729880 1 bucket sync caught up with source: 2023-09-29T23:45:04.143 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [00000000001.54.6, , 00000000001.32.6, 00000000001.31.6, 00000000001.43.6, 00000000001.48.6, 00000000001.47.6, 00000000001.51.6, 00000000001.28.6, 00000000001.30.6, 00000000001.52.6] 2023-09-29T23:45:04.143 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [00000000001.54.6, , 00000000001.32.6, 00000000001.31.6, 00000000001.43.6, 00000000001.48.6, 00000000001.47.6, 00000000001.51.6, 00000000001.28.6, 00000000001.30.6, 00000000001.52.6] 2023-09-29T23:45:04.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:04.142+0000 7f33d5729880 0 bucket checkpoint complete 2023-09-29T23:45:04.161 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-69 2023-09-29T23:45:04.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:45:04.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:45:04.161 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:45:04.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:45:04.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:45:04.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:04.270+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:09.270 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:09.270+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:14.271 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:14.271+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:19.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:19.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:24.272 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:24.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:29.273 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:29.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:34.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:34.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:39.274 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:39.273+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:44.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:44.274+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:49.275 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:49.275+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:54.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:54.276+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:45:59.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:45:59.276+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:04.277 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:04.277+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:09.278 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:09.277+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:14.279 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:14.279+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:19.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:19.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:24.280 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:24.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:29.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:29.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:34.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:34.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:39.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:39.281+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:44.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:44.282+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:49.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:49.283+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:54.283 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:54.283+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:46:59.284 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:46:59.284+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:04.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:04.284+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:09.285 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:09.285+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:14.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:14.286+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:19.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:19.286+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:24.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:24.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:29.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:29.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:34.288 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:34.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:39.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:39.288+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:44.289 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:44.289+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:49.290 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:49.290+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:54.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:54.291+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:47:59.291 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:47:59.291+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:04.292 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:04.292+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:09.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:09.292+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:14.293 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:14.293+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:19.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:19.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:24.294 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:24.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:29.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:29.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:34.295 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:34.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:39.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:39.295+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:44.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:44.296+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:49.297 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:49.297+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:54.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:54.297+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:48:59.298 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:48:59.298+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:04.299 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:04.299+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:09.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:09.299+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:14.300 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:14.300+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:19.301 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:19.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:24.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:24.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:29.302 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:29.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:34.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:34.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:39.303 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:39.303+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:44.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:44.304+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:49.304 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:49.304+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:54.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:54.305+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-29T23:49:59.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:59.305+0000 7fc112612880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:49:59.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:59.305+0000 7fc112612880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:49:59.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:59.305+0000 7fc112612880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:49:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:49:59.325 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2023-09-29T23:49:59.326 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.326 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.326 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.326 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.327 DEBUG:boto:Data: 2023-09-29T23:49:59.327 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.327 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.327 DEBUG:boto:Params: {} 2023-09-29T23:49:59.327 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:49:59.327 DEBUG:boto:Token: None 2023-09-29T23:49:59.328 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UMj4/X0lYaJ59XdpOKK7iRzp9ac= 2023-09-29T23:49:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UMj4/X0lYaJ59XdpOKK7iRzp9ac=', 'Content-Length': '0'} 2023-09-29T23:49:59.351 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4446aff7e8b68a2-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.351 DEBUG:rgw_multi.tests:created bucket=dmuhsu-70 2023-09-29T23:49:59.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.352 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:49:59.408 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:49:59.408 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:49:59.444 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031399.349060_574.1", 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:49:59.349060Z" 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:49:59.464 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.465 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:49:59.465 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:49:59.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:49:59.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.466 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:49:59.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:49:59.520 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:49:59.557 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:49:59.556+0000 7f93a6711880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:49:59.557 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:49:59.557+0000 7f93a6711880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:49:59.571 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:49:59.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:49:59.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2023-09-29T23:49:59.574 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:49:59.575 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031399.349060_574.1", 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:49:59.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:49:59.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:49:59Z" 2023-09-29T23:49:59.578 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:49:59.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031399.349060_574.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:49:59.349060Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:49:59Z" } 2023-09-29T23:49:59.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031399.349060_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:49:59.349060Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:49:59.580 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:49:59.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:49:59.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:49:59.580 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:49:59.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:49:59.581 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:49:59.627 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:49:59.628 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:49:59.679 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:59.679+0000 7fcea180f880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:49:59.680 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:49:59.680+0000 7fcea180f880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:49:59.702 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025381.280108_31.1", 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:49:59.703 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:49:59.704 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:49:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031399.349060_574.1", 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:49:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:49:59.707 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:49:59Z" 2023-09-29T23:49:59.708 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:49:59.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696025381.280108_31.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:09:41.280108Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031399.349060_574.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:49:59.349060Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:49:59Z" } 2023-09-29T23:49:59.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031399.349060_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:49:59.349060Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:49:59.710 DEBUG:rgw_multi.tests:log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:49:59.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:49:59.710 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:49:59.710 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.710 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.710 DEBUG:boto:Method: HEAD 2023-09-29T23:49:59.710 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.710 DEBUG:boto:Data: 2023-09-29T23:49:59.711 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.711 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.711 DEBUG:boto:Params: {} 2023-09-29T23:49:59.711 DEBUG:boto:Token: None 2023-09-29T23:49:59.711 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.711 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-29T23:49:59.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-29T23:49:59.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020cfaf7a42d4cc37-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.714 DEBUG:boto:path=/dmuhsu-70/obj11 2023-09-29T23:49:59.714 DEBUG:boto:auth_path=/dmuhsu-70/obj11 2023-09-29T23:49:59.714 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.714 DEBUG:boto:Path: /dmuhsu-70/obj11 2023-09-29T23:49:59.714 DEBUG:boto:Data: 2023-09-29T23:49:59.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:49:59.715 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.715 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.715 DEBUG:boto:Params: {} 2023-09-29T23:49:59.715 DEBUG:boto:Token: None 2023-09-29T23:49:59.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/obj11 2023-09-29T23:49:59.715 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qox4yuk+gZKccobO3tqCvYwjmf0= 2023-09-29T23:49:59.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qox4yuk+gZKccobO3tqCvYwjmf0='} 2023-09-29T23:49:59.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbf8beea3f9c4f59-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.724 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.724 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.724 DEBUG:boto:Method: HEAD 2023-09-29T23:49:59.724 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.725 DEBUG:boto:Data: 2023-09-29T23:49:59.725 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.725 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.725 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.725 DEBUG:boto:Params: {} 2023-09-29T23:49:59.725 DEBUG:boto:Token: None 2023-09-29T23:49:59.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.725 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-29T23:49:59.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-29T23:49:59.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbb293bf633d25b2-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.728 DEBUG:boto:path=/dmuhsu-70/obj22 2023-09-29T23:49:59.728 DEBUG:boto:auth_path=/dmuhsu-70/obj22 2023-09-29T23:49:59.729 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.729 DEBUG:boto:Path: /dmuhsu-70/obj22 2023-09-29T23:49:59.729 DEBUG:boto:Data: 2023-09-29T23:49:59.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:49:59.729 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.729 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.729 DEBUG:boto:Params: {} 2023-09-29T23:49:59.729 DEBUG:boto:Token: None 2023-09-29T23:49:59.730 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/obj22 2023-09-29T23:49:59.730 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DVUNWzUT2oM6Z6N7m0EpBcQzg6M= 2023-09-29T23:49:59.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DVUNWzUT2oM6Z6N7m0EpBcQzg6M='} 2023-09-29T23:49:59.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021fd4751eeb5f837-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.735 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.735 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.735 DEBUG:boto:Method: HEAD 2023-09-29T23:49:59.735 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.735 DEBUG:boto:Data: 2023-09-29T23:49:59.736 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.736 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.736 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.736 DEBUG:boto:Params: {} 2023-09-29T23:49:59.736 DEBUG:boto:Token: None 2023-09-29T23:49:59.736 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-29T23:49:59.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-29T23:49:59.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e2c01ab3d94a544-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.738 DEBUG:boto:path=/dmuhsu-70/obj33 2023-09-29T23:49:59.739 DEBUG:boto:auth_path=/dmuhsu-70/obj33 2023-09-29T23:49:59.739 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.739 DEBUG:boto:Path: /dmuhsu-70/obj33 2023-09-29T23:49:59.739 DEBUG:boto:Data: 2023-09-29T23:49:59.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:49:59.739 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.739 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.739 DEBUG:boto:Params: {} 2023-09-29T23:49:59.739 DEBUG:boto:Token: None 2023-09-29T23:49:59.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/obj33 2023-09-29T23:49:59.740 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lnb8t+vBhgdbKeIJNV2CNpcZ38k= 2023-09-29T23:49:59.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lnb8t+vBhgdbKeIJNV2CNpcZ38k='} 2023-09-29T23:49:59.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080d40c7c3fb15f87-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.787 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.787 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.787 DEBUG:boto:Method: HEAD 2023-09-29T23:49:59.787 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.787 DEBUG:boto:Data: 2023-09-29T23:49:59.787 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.787 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.787 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.788 DEBUG:boto:Params: {} 2023-09-29T23:49:59.788 DEBUG:boto:Token: None 2023-09-29T23:49:59.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.788 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-29T23:49:59.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-29T23:49:59.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b49374d7276c918f-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.790 DEBUG:boto:path=/dmuhsu-70/obj44 2023-09-29T23:49:59.791 DEBUG:boto:auth_path=/dmuhsu-70/obj44 2023-09-29T23:49:59.791 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.791 DEBUG:boto:Path: /dmuhsu-70/obj44 2023-09-29T23:49:59.791 DEBUG:boto:Data: 2023-09-29T23:49:59.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:49:59.791 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.791 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.791 DEBUG:boto:Params: {} 2023-09-29T23:49:59.791 DEBUG:boto:Token: None 2023-09-29T23:49:59.792 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/obj44 2023-09-29T23:49:59.792 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hYgpFD6UYowD19KsUosHXZRuiFA= 2023-09-29T23:49:59.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hYgpFD6UYowD19KsUosHXZRuiFA='} 2023-09-29T23:49:59.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005804958fc9dd4c81-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.801 DEBUG:boto:path=/dmuhsu-70/ 2023-09-29T23:49:59.801 DEBUG:boto:auth_path=/dmuhsu-70/ 2023-09-29T23:49:59.801 DEBUG:boto:Method: HEAD 2023-09-29T23:49:59.801 DEBUG:boto:Path: /dmuhsu-70/ 2023-09-29T23:49:59.801 DEBUG:boto:Data: 2023-09-29T23:49:59.801 DEBUG:boto:Headers: {} 2023-09-29T23:49:59.801 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.801 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.802 DEBUG:boto:Params: {} 2023-09-29T23:49:59.802 DEBUG:boto:Token: None 2023-09-29T23:49:59.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/ 2023-09-29T23:49:59.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-29T23:49:59.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-29T23:49:59.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295da537e09e17ea-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.804 DEBUG:boto:path=/dmuhsu-70/obj55 2023-09-29T23:49:59.804 DEBUG:boto:auth_path=/dmuhsu-70/obj55 2023-09-29T23:49:59.805 DEBUG:boto:Method: PUT 2023-09-29T23:49:59.805 DEBUG:boto:Path: /dmuhsu-70/obj55 2023-09-29T23:49:59.805 DEBUG:boto:Data: 2023-09-29T23:49:59.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:49:59.805 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:49:59.805 DEBUG:boto:Port: 8000 2023-09-29T23:49:59.805 DEBUG:boto:Params: {} 2023-09-29T23:49:59.805 DEBUG:boto:Token: None 2023-09-29T23:49:59.805 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:49:59 GMT /dmuhsu-70/obj55 2023-09-29T23:49:59.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:lqyyIpJKCuIY/PkxBOe4yG4EZWE= 2023-09-29T23:49:59.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lqyyIpJKCuIY/PkxBOe4yG4EZWE='} 2023-09-29T23:49:59.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b8915339b7a1aed-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:49:59.855 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-70 2023-09-29T23:49:59.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:49:59.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:49:59.855 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:49:59.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:49:59.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:49:59.943 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-70 2023-09-29T23:49:59.958 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-70 2023-09-29T23:49:59.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:49:59.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:49:59.958 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:50:00.010 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:50:00.010 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:50:00.067 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:00.067+0000 7f888649d880 1 waiting to reach incremental sync.. 2023-09-29T23:50:05.069 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:05.069+0000 7f888649d880 1 bucket sync caught up with source: 2023-09-29T23:50:05.070 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.35.6, 00000000002.23.6, 00000000002.58.6] 2023-09-29T23:50:05.070 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.35.6, 00000000002.23.6, 00000000002.58.6] 2023-09-29T23:50:05.070 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:05.069+0000 7f888649d880 0 bucket checkpoint complete 2023-09-29T23:50:05.092 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-70 2023-09-29T23:50:05.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:50:05.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:50:05.093 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:50:05.150 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:50:05.150 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:50:05.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:05.204+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:10.204 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:10.204+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:15.205 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:15.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:20.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:20.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:25.206 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:25.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:30.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:30.206+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:35.207 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:35.206+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:40.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:40.207+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:45.208 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:45.208+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:50.209 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:50.209+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:50:55.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:50:55.209+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:00.210 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:00.210+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:05.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:05.210+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:10.211 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:10.211+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:15.212 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:15.212+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:20.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:20.213+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:25.213 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:25.212+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:30.214 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:30.213+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:35.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:35.214+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:40.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:40.215+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:45.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:45.216+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:50.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:50.216+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:51:55.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:51:55.217+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:00.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:00.217+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:05.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:05.218+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:10.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:10.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:15.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:15.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:20.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:20.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:25.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:25.220+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:30.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:30.220+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:35.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:35.221+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:40.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:40.222+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:45.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:45.223+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:50.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:50.223+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:52:55.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:52:55.224+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:00.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:00.224+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:05.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:05.225+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:10.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:10.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:15.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:15.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:20.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:20.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:25.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:30.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:30.227+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:35.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:35.229+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:40.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:40.229+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:45.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:45.230+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:50.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:50.230+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:53:55.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:53:55.231+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:00.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:00.231+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:05.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:05.232+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:10.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:10.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:15.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:15.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:20.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:20.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:25.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:25.234+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:30.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:30.234+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:35.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:35.235+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:40.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:40.236+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:45.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:45.237+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:50.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:50.237+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:54:55.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:54:55.238+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-29T23:55:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:00.238+0000 7f5634e0f880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:55:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:00.238+0000 7f5634e0f880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:55:00.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:00.238+0000 7f5634e0f880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:55:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:55:00.261 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2023-09-29T23:55:00.262 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-dmuhsu-1 2023-09-29T23:55:00.263 DEBUG:boto:Method: POST 2023-09-29T23:55:00.263 DEBUG:boto:Path: / 2023-09-29T23:55:00.263 DEBUG:boto:Data: 2023-09-29T23:55:00.263 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.263 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.263 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.263 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.263 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:55:00.264 DEBUG:boto:Token: None 2023-09-29T23:55:00.264 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date 63f62b5affd288c2bef45ff2a92621012fd5d830dfdab301910c3efb90ef85d2 2023-09-29T23:55:00.264 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request bb587c2c7cbc183d251da6426a41c8b200e8c7da49f0af787ff0fe5b6b1bcc00 2023-09-29T23:55:00.264 DEBUG:boto:Signature: c18775c3e3788e2b8cfd0e796b13fbd9ba096c3940f4889547b3af3db64e310c 2023-09-29T23:55:00.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=c18775c3e3788e2b8cfd0e796b13fbd9ba096c3940f4889547b3af3db64e310c', 'Host': 'smithi175.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.280 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d6c76d9a22c5942-00651763d4-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.281 DEBUG:boto:b'd750ed68-7545-41e2-9df6-70b73259c4b1roles-dmuhsu-1/arn:aws:iam:::role/roles-dmuhsu-12023-09-29T23:55:00.269Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000002d6c76d9a22c5942-00651763d4-1e9c-test-zone1' 2023-09-29T23:55:00.281 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-dmuhsu-2 2023-09-29T23:55:00.281 DEBUG:boto:Method: POST 2023-09-29T23:55:00.281 DEBUG:boto:Path: / 2023-09-29T23:55:00.282 DEBUG:boto:Data: 2023-09-29T23:55:00.282 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.282 DEBUG:boto:Host: smithi184.front.sepia.ceph.com 2023-09-29T23:55:00.282 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.282 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.282 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:55:00.282 DEBUG:boto:Token: None 2023-09-29T23:55:00.282 DEBUG:boto:CanonicalRequest: POST / host:smithi184.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date dc005220f478eba63ba77228d19d71dc022254fb5d036db8d91d45a8f1ba6b51 2023-09-29T23:55:00.283 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi184/aws4_request 3a8e431f15da628b923b9f4f96e9e00660b69c564e63b230d0bad26164a5e393 2023-09-29T23:55:00.283 DEBUG:boto:Signature: 420461cdb09436166c51177ddc757cdb2c724694a6c21821ea43d20edaa0adbc 2023-09-29T23:55:00.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi184/aws4_request,SignedHeaders=host;x-amz-date,Signature=420461cdb09436166c51177ddc757cdb2c724694a6c21821ea43d20edaa0adbc', 'Host': 'smithi184.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.309 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b36b5037c72d3ba-00651763d4-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.309 DEBUG:boto:b'39a8f554-6745-4a79-aaa8-b62dc5785d98roles-dmuhsu-2/arn:aws:iam:::role/roles-dmuhsu-22023-09-29T23:55:00.303Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000006b36b5037c72d3ba-00651763d4-18a8-test-zone2' 2023-09-29T23:55:00.310 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-dmuhsu-3 2023-09-29T23:55:00.310 DEBUG:boto:Method: POST 2023-09-29T23:55:00.311 DEBUG:boto:Path: / 2023-09-29T23:55:00.311 DEBUG:boto:Data: 2023-09-29T23:55:00.311 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.311 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.311 DEBUG:boto:Port: 8001 2023-09-29T23:55:00.311 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.311 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:55:00.311 DEBUG:boto:Token: None 2023-09-29T23:55:00.311 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8001 x-amz-date:20230929T235500Z host;x-amz-date be546b9d99f4d65d73a44c41192b01353ce8aa4770b29a92e1c526315b931f70 2023-09-29T23:55:00.312 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request 6d8a4eefb16f341fb6b09a25d483385f140f4b3ffdb9a3dfb4379384efdb68db 2023-09-29T23:55:00.312 DEBUG:boto:Signature: 4e4d56ded6eb506ffb9e193109dc50387c5bf6fa0b60912afe6c0a628151cad9 2023-09-29T23:55:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=4e4d56ded6eb506ffb9e193109dc50387c5bf6fa0b60912afe6c0a628151cad9', 'Host': 'smithi175.front.sepia.ceph.com:8001'} 2023-09-29T23:55:00.334 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003dc57a4068a51acd-00651763d4-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.334 DEBUG:boto:b'f73c46ab-386f-46c3-a2ee-b1b98cd08ed1roles-dmuhsu-3/arn:aws:iam:::role/roles-dmuhsu-32023-09-29T23:55:00.328Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000003dc57a4068a51acd-00651763d4-1fde-test-zone3' 2023-09-29T23:55:00.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.335 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:00.395 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:00.395 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:00.436 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:55:00.459 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:55:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.459 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.322172_577.1", 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.322172Z" 2023-09-29T23:55:00.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:55:00.461 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.461 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:55:00.462 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:55:00.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.463 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:00.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:00.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:00.552 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:00.552+0000 7fb53d612880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.553 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:00.553+0000 7fb53d612880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.567 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:55:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:55:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:55:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.322172_577.1", 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.322172Z", 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:55:00Z" 2023-09-29T23:55:00.574 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:55:00.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031700.322172_577.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.322172Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:55:00Z" } 2023-09-29T23:55:00.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.322172_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.322172Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:55:00.576 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:55:00.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:55:00.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.577 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:00.622 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:00.622 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:00.654 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:00.654+0000 7f8086067880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.655 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:00.655+0000 7f8086067880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.673 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:55:00.673 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:55:00.673 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:55:00.673 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:55:00.673 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.674 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:55:00.675 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:55:00.676 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:00.677 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.678 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.678 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.678 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:55:00.678 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:00.678 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.322172_577.1", 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.322172Z", 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.679 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:00.680 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:55:00.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.682 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:55:00Z" 2023-09-29T23:55:00.682 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:55:00.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031700.322172_577.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.322172Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:55:00Z" } 2023-09-29T23:55:00.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.322172_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.322172Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:55:00.683 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.684 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:55:00.684 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-1 zones={test-zone2, test-zone1} 2023-09-29T23:55:00.684 DEBUG:boto:Method: POST 2023-09-29T23:55:00.684 DEBUG:boto:Path: / 2023-09-29T23:55:00.684 DEBUG:boto:Data: 2023-09-29T23:55:00.684 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.684 DEBUG:boto:Host: smithi184.front.sepia.ceph.com 2023-09-29T23:55:00.684 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.684 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.684 DEBUG:boto:Token: None 2023-09-29T23:55:00.685 DEBUG:boto:CanonicalRequest: POST / host:smithi184.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date 927c8996a40bda51ec3e2dbf6ed4df293ced2f2070a1fe3064eae9b71bb7d216 2023-09-29T23:55:00.685 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi184/aws4_request d36af2a5a5b8dd736e95f8b06c1dbdf0903254d682f42be830392cb1a402cf52 2023-09-29T23:55:00.685 DEBUG:boto:Signature: 05974ca25b91b4972eccc0a1a55fdf0e902b440ab663f1953c4435e068fe0b81 2023-09-29T23:55:00.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi184/aws4_request,SignedHeaders=host;x-amz-date,Signature=05974ca25b91b4972eccc0a1a55fdf0e902b440ab663f1953c4435e068fe0b81', 'Host': 'smithi184.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.687 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bca2f71deb92975a-00651763d4-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.687 DEBUG:boto:b'tx00000bca2f71deb92975a-00651763d4-18a8-test-zone2d750ed68-7545-41e2-9df6-70b73259c4b1roles-dmuhsu-1/arn:aws:iam:::role/roles-dmuhsu-12023-09-29T23:55:00.478Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.688 DEBUG:boto:Method: POST 2023-09-29T23:55:00.688 DEBUG:boto:Path: / 2023-09-29T23:55:00.688 DEBUG:boto:Data: 2023-09-29T23:55:00.688 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.688 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.688 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.688 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.689 DEBUG:boto:Token: None 2023-09-29T23:55:00.689 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date 927c8996a40bda51ec3e2dbf6ed4df293ced2f2070a1fe3064eae9b71bb7d216 2023-09-29T23:55:00.689 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request a06b7b469ddc617b303d4f3dd8890e0104371fed55507b668d6b0e65651a5cda 2023-09-29T23:55:00.689 DEBUG:boto:Signature: 046e335ea25c301fa70062fdaf2b37aa9b93a7fcdea78877f2d2d7e779fbc2fc 2023-09-29T23:55:00.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=046e335ea25c301fa70062fdaf2b37aa9b93a7fcdea78877f2d2d7e779fbc2fc', 'Host': 'smithi175.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.690 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cda77b2e267588bb-00651763d4-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.691 DEBUG:boto:b'tx00000cda77b2e267588bb-00651763d4-1e9c-test-zone1d750ed68-7545-41e2-9df6-70b73259c4b1roles-dmuhsu-1/arn:aws:iam:::role/roles-dmuhsu-12023-09-29T23:55:00.269Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.691 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-1 2023-09-29T23:55:00.691 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-1 zones={test-zone2, test-zone1} 2023-09-29T23:55:00.691 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-1 zones={test-zone3, test-zone1} 2023-09-29T23:55:00.692 DEBUG:boto:Method: POST 2023-09-29T23:55:00.692 DEBUG:boto:Path: / 2023-09-29T23:55:00.692 DEBUG:boto:Data: 2023-09-29T23:55:00.692 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.692 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.692 DEBUG:boto:Port: 8001 2023-09-29T23:55:00.693 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.693 DEBUG:boto:Token: None 2023-09-29T23:55:00.693 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8001 x-amz-date:20230929T235500Z host;x-amz-date 927c8996a40bda51ec3e2dbf6ed4df293ced2f2070a1fe3064eae9b71bb7d216 2023-09-29T23:55:00.693 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request 18d9070a4ecf6dea2a74b017d22e4380f91f345440a936f932e0056be0954bab 2023-09-29T23:55:00.693 DEBUG:boto:Signature: d146983dc4dd5aa1b83ede103c3095dbccc95d99a0b8d4703d1297cc537f3cf3 2023-09-29T23:55:00.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=d146983dc4dd5aa1b83ede103c3095dbccc95d99a0b8d4703d1297cc537f3cf3', 'Host': 'smithi175.front.sepia.ceph.com:8001'} 2023-09-29T23:55:00.696 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d049d7057e635e3a-00651763d4-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.696 DEBUG:boto:b'tx00000d049d7057e635e3a-00651763d4-1fde-test-zone3d750ed68-7545-41e2-9df6-70b73259c4b1roles-dmuhsu-1/arn:aws:iam:::role/roles-dmuhsu-12023-09-29T23:55:00.476Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.696 DEBUG:boto:Method: POST 2023-09-29T23:55:00.696 DEBUG:boto:Path: / 2023-09-29T23:55:00.697 DEBUG:boto:Data: 2023-09-29T23:55:00.697 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.697 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.697 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.697 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.697 DEBUG:boto:Token: None 2023-09-29T23:55:00.698 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date 927c8996a40bda51ec3e2dbf6ed4df293ced2f2070a1fe3064eae9b71bb7d216 2023-09-29T23:55:00.698 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request a06b7b469ddc617b303d4f3dd8890e0104371fed55507b668d6b0e65651a5cda 2023-09-29T23:55:00.698 DEBUG:boto:Signature: 046e335ea25c301fa70062fdaf2b37aa9b93a7fcdea78877f2d2d7e779fbc2fc 2023-09-29T23:55:00.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=046e335ea25c301fa70062fdaf2b37aa9b93a7fcdea78877f2d2d7e779fbc2fc', 'Host': 'smithi175.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9c796a835cd467e-00651763d4-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.700 DEBUG:boto:b'tx00000e9c796a835cd467e-00651763d4-1e9c-test-zone1d750ed68-7545-41e2-9df6-70b73259c4b1roles-dmuhsu-1/arn:aws:iam:::role/roles-dmuhsu-12023-09-29T23:55:00.269Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.700 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-1 2023-09-29T23:55:00.700 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-1 zones={test-zone3, test-zone1} 2023-09-29T23:55:00.700 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-2 zones={test-zone1, test-zone2} 2023-09-29T23:55:00.701 DEBUG:boto:Method: POST 2023-09-29T23:55:00.701 DEBUG:boto:Path: / 2023-09-29T23:55:00.701 DEBUG:boto:Data: 2023-09-29T23:55:00.701 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.701 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.701 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.702 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.702 DEBUG:boto:Token: None 2023-09-29T23:55:00.702 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date b8e8fd3f867a7b0a6b2e5c1e9043d0bce10d74b19bdc1ad556ffe559f9228bb8 2023-09-29T23:55:00.702 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request e6020363b1f60987e833152809418eb81f653caec27812a8d9c28e4da0656833 2023-09-29T23:55:00.703 DEBUG:boto:Signature: 5fdbfba867ed40664e2f66ce7a8e6b8b395a2ac4d14ef71e60c8f8db01fbf319 2023-09-29T23:55:00.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=5fdbfba867ed40664e2f66ce7a8e6b8b395a2ac4d14ef71e60c8f8db01fbf319', 'Host': 'smithi175.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.704 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056758d9ca5736404-00651763d4-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.704 DEBUG:boto:b'tx0000056758d9ca5736404-00651763d4-1e9c-test-zone139a8f554-6745-4a79-aaa8-b62dc5785d98roles-dmuhsu-2/arn:aws:iam:::role/roles-dmuhsu-22023-09-29T23:55:00.289Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.705 DEBUG:boto:Method: POST 2023-09-29T23:55:00.705 DEBUG:boto:Path: / 2023-09-29T23:55:00.705 DEBUG:boto:Data: 2023-09-29T23:55:00.705 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.705 DEBUG:boto:Host: smithi184.front.sepia.ceph.com 2023-09-29T23:55:00.705 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.705 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.706 DEBUG:boto:Token: None 2023-09-29T23:55:00.706 DEBUG:boto:CanonicalRequest: POST / host:smithi184.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date b8e8fd3f867a7b0a6b2e5c1e9043d0bce10d74b19bdc1ad556ffe559f9228bb8 2023-09-29T23:55:00.706 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi184/aws4_request 9397edebae978e094d68379250ab601bc87ad7dc266ac8b57e7235db4ee10439 2023-09-29T23:55:00.706 DEBUG:boto:Signature: 8ea2b37756b580bdbef50252fbc9c13ac26d6d84a199ed1ab0d654cdcc0d0ec5 2023-09-29T23:55:00.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi184/aws4_request,SignedHeaders=host;x-amz-date,Signature=8ea2b37756b580bdbef50252fbc9c13ac26d6d84a199ed1ab0d654cdcc0d0ec5', 'Host': 'smithi184.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.708 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000694b4536742e9980-00651763d4-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.708 DEBUG:boto:b'tx00000694b4536742e9980-00651763d4-18a8-test-zone239a8f554-6745-4a79-aaa8-b62dc5785d98roles-dmuhsu-2/arn:aws:iam:::role/roles-dmuhsu-22023-09-29T23:55:00.289Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.709 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-2 2023-09-29T23:55:00.709 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-2 zones={test-zone1, test-zone2} 2023-09-29T23:55:00.709 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-2 zones={test-zone3, test-zone2} 2023-09-29T23:55:00.709 DEBUG:boto:Method: POST 2023-09-29T23:55:00.709 DEBUG:boto:Path: / 2023-09-29T23:55:00.709 DEBUG:boto:Data: 2023-09-29T23:55:00.710 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.710 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.710 DEBUG:boto:Port: 8001 2023-09-29T23:55:00.710 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.710 DEBUG:boto:Token: None 2023-09-29T23:55:00.711 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8001 x-amz-date:20230929T235500Z host;x-amz-date b8e8fd3f867a7b0a6b2e5c1e9043d0bce10d74b19bdc1ad556ffe559f9228bb8 2023-09-29T23:55:00.711 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request b74ba0eca788ad89fe43103d74879e6645e5d6bf7fae718ad2bff75d7f41d602 2023-09-29T23:55:00.711 DEBUG:boto:Signature: a27e6d58ac8ee5b3b37046b3e89d4612aca292680495df318a80159e59039d63 2023-09-29T23:55:00.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=a27e6d58ac8ee5b3b37046b3e89d4612aca292680495df318a80159e59039d63', 'Host': 'smithi175.front.sepia.ceph.com:8001'} 2023-09-29T23:55:00.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008df025ea6ad4526b-00651763d4-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.713 DEBUG:boto:b'tx000008df025ea6ad4526b-00651763d4-1fde-test-zone339a8f554-6745-4a79-aaa8-b62dc5785d98roles-dmuhsu-2/arn:aws:iam:::role/roles-dmuhsu-22023-09-29T23:55:00.481Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.713 DEBUG:boto:Method: POST 2023-09-29T23:55:00.714 DEBUG:boto:Path: / 2023-09-29T23:55:00.714 DEBUG:boto:Data: 2023-09-29T23:55:00.714 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.714 DEBUG:boto:Host: smithi184.front.sepia.ceph.com 2023-09-29T23:55:00.714 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.714 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.715 DEBUG:boto:Token: None 2023-09-29T23:55:00.715 DEBUG:boto:CanonicalRequest: POST / host:smithi184.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date b8e8fd3f867a7b0a6b2e5c1e9043d0bce10d74b19bdc1ad556ffe559f9228bb8 2023-09-29T23:55:00.715 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi184/aws4_request 9397edebae978e094d68379250ab601bc87ad7dc266ac8b57e7235db4ee10439 2023-09-29T23:55:00.715 DEBUG:boto:Signature: 8ea2b37756b580bdbef50252fbc9c13ac26d6d84a199ed1ab0d654cdcc0d0ec5 2023-09-29T23:55:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi184/aws4_request,SignedHeaders=host;x-amz-date,Signature=8ea2b37756b580bdbef50252fbc9c13ac26d6d84a199ed1ab0d654cdcc0d0ec5', 'Host': 'smithi184.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.716 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e8a025cf31cb40b-00651763d4-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.717 DEBUG:boto:b'tx000009e8a025cf31cb40b-00651763d4-18a8-test-zone239a8f554-6745-4a79-aaa8-b62dc5785d98roles-dmuhsu-2/arn:aws:iam:::role/roles-dmuhsu-22023-09-29T23:55:00.289Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.717 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-2 2023-09-29T23:55:00.717 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-2 zones={test-zone3, test-zone2} 2023-09-29T23:55:00.717 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-3 zones={test-zone1, test-zone3} 2023-09-29T23:55:00.718 DEBUG:boto:Method: POST 2023-09-29T23:55:00.718 DEBUG:boto:Path: / 2023-09-29T23:55:00.718 DEBUG:boto:Data: 2023-09-29T23:55:00.718 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.718 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.718 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.718 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.719 DEBUG:boto:Token: None 2023-09-29T23:55:00.719 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date f17e0e22a0496708d36453bf29d28ea7c4197a097ab96eb444ef9c29a31b1e85 2023-09-29T23:55:00.719 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request d187fb064dc0d74a9c40d82819ae297ba66bbdad0be19a55bc31dbe7ea395bc4 2023-09-29T23:55:00.719 DEBUG:boto:Signature: 3d7c7a9da9788369b9d14cf8b679d15672fdbd96a0d332812370dc62172d7fb7 2023-09-29T23:55:00.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=3d7c7a9da9788369b9d14cf8b679d15672fdbd96a0d332812370dc62172d7fb7', 'Host': 'smithi175.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.721 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000470f58d964e84adb-00651763d4-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.721 DEBUG:boto:b'tx00000470f58d964e84adb-00651763d4-1e9c-test-zone1f73c46ab-386f-46c3-a2ee-b1b98cd08ed1roles-dmuhsu-3/arn:aws:iam:::role/roles-dmuhsu-32023-09-29T23:55:00.317Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.721 DEBUG:boto:Method: POST 2023-09-29T23:55:00.721 DEBUG:boto:Path: / 2023-09-29T23:55:00.722 DEBUG:boto:Data: 2023-09-29T23:55:00.722 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.722 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.722 DEBUG:boto:Port: 8001 2023-09-29T23:55:00.722 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.723 DEBUG:boto:Token: None 2023-09-29T23:55:00.723 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8001 x-amz-date:20230929T235500Z host;x-amz-date f17e0e22a0496708d36453bf29d28ea7c4197a097ab96eb444ef9c29a31b1e85 2023-09-29T23:55:00.723 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request fe8ee059433380f45deade51e7e29d8a27564a349240df4cfdb3a5d98176bdab 2023-09-29T23:55:00.723 DEBUG:boto:Signature: bb4b525e103742322cc8215d2bdfd7aa0c32a10427603b0ecf6c7e72d983511f 2023-09-29T23:55:00.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb4b525e103742322cc8215d2bdfd7aa0c32a10427603b0ecf6c7e72d983511f', 'Host': 'smithi175.front.sepia.ceph.com:8001'} 2023-09-29T23:55:00.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008652d650b79f7a17-00651763d4-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.725 DEBUG:boto:b'tx000008652d650b79f7a17-00651763d4-1fde-test-zone3f73c46ab-386f-46c3-a2ee-b1b98cd08ed1roles-dmuhsu-3/arn:aws:iam:::role/roles-dmuhsu-32023-09-29T23:55:00.317Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.726 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-3 2023-09-29T23:55:00.726 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-3 zones={test-zone1, test-zone3} 2023-09-29T23:55:00.726 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dmuhsu-3 zones={test-zone2, test-zone3} 2023-09-29T23:55:00.726 DEBUG:boto:Method: POST 2023-09-29T23:55:00.726 DEBUG:boto:Path: / 2023-09-29T23:55:00.727 DEBUG:boto:Data: 2023-09-29T23:55:00.727 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.727 DEBUG:boto:Host: smithi184.front.sepia.ceph.com 2023-09-29T23:55:00.727 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.727 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.727 DEBUG:boto:Token: None 2023-09-29T23:55:00.728 DEBUG:boto:CanonicalRequest: POST / host:smithi184.front.sepia.ceph.com:8000 x-amz-date:20230929T235500Z host;x-amz-date f17e0e22a0496708d36453bf29d28ea7c4197a097ab96eb444ef9c29a31b1e85 2023-09-29T23:55:00.728 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi184/aws4_request 07727d8c77b5a124ca5e4563be4afb8e72876955de3e25e604985111dd65a1cb 2023-09-29T23:55:00.728 DEBUG:boto:Signature: 65214f2708331f04ed5f49a6e5b1790258125a676cf63c7217ea9a18b482f9e4 2023-09-29T23:55:00.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi184/aws4_request,SignedHeaders=host;x-amz-date,Signature=65214f2708331f04ed5f49a6e5b1790258125a676cf63c7217ea9a18b482f9e4', 'Host': 'smithi184.front.sepia.ceph.com:8000'} 2023-09-29T23:55:00.730 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7c8eb511081d5d4-00651763d4-18a8-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.730 DEBUG:boto:b'tx00000c7c8eb511081d5d4-00651763d4-18a8-test-zone2f73c46ab-386f-46c3-a2ee-b1b98cd08ed1roles-dmuhsu-3/arn:aws:iam:::role/roles-dmuhsu-32023-09-29T23:55:00.480Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.731 DEBUG:boto:Method: POST 2023-09-29T23:55:00.731 DEBUG:boto:Path: / 2023-09-29T23:55:00.731 DEBUG:boto:Data: 2023-09-29T23:55:00.731 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com 2023-09-29T23:55:00.731 DEBUG:boto:Port: 8001 2023-09-29T23:55:00.731 DEBUG:boto:Params: {'RoleName': 'roles-dmuhsu-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T23:55:00.731 DEBUG:boto:Token: None 2023-09-29T23:55:00.732 DEBUG:boto:CanonicalRequest: POST / host:smithi175.front.sepia.ceph.com:8001 x-amz-date:20230929T235500Z host;x-amz-date f17e0e22a0496708d36453bf29d28ea7c4197a097ab96eb444ef9c29a31b1e85 2023-09-29T23:55:00.732 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T235500Z 20230929/front/smithi175/aws4_request fe8ee059433380f45deade51e7e29d8a27564a349240df4cfdb3a5d98176bdab 2023-09-29T23:55:00.732 DEBUG:boto:Signature: bb4b525e103742322cc8215d2bdfd7aa0c32a10427603b0ecf6c7e72d983511f 2023-09-29T23:55:00.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T235500Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=PTXB0TDQIX657E2O5ERJ/20230929/front/smithi175/aws4_request,SignedHeaders=host;x-amz-date,Signature=bb4b525e103742322cc8215d2bdfd7aa0c32a10427603b0ecf6c7e72d983511f', 'Host': 'smithi175.front.sepia.ceph.com:8001'} 2023-09-29T23:55:00.733 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abb8caefb345f50f-00651763d4-1fde-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.733 DEBUG:boto:b'tx00000abb8caefb345f50f-00651763d4-1fde-test-zone3f73c46ab-386f-46c3-a2ee-b1b98cd08ed1roles-dmuhsu-3/arn:aws:iam:::role/roles-dmuhsu-32023-09-29T23:55:00.317Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T23:55:00.734 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dmuhsu-3 2023-09-29T23:55:00.734 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dmuhsu-3 zones={test-zone2, test-zone3} 2023-09-29T23:55:00.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-09-29T23:55:00.734 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:00.735 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:00.735 DEBUG:boto:Method: PUT 2023-09-29T23:55:00.735 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:00.735 DEBUG:boto:Data: 2023-09-29T23:55:00.735 DEBUG:boto:Headers: {} 2023-09-29T23:55:00.735 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:00.735 DEBUG:boto:Port: 8000 2023-09-29T23:55:00.735 DEBUG:boto:Params: {} 2023-09-29T23:55:00.736 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:55:00.736 DEBUG:boto:Token: None 2023-09-29T23:55:00.736 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:55:00 GMT /dmuhsu-71/ 2023-09-29T23:55:00.736 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FPpaQWWHx9DGoJL8l4pa0soUFX8= 2023-09-29T23:55:00.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FPpaQWWHx9DGoJL8l4pa0soUFX8=', 'Content-Length': '0'} 2023-09-29T23:55:00.763 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ceb52896743f036-00651763d4-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:00.763 DEBUG:rgw_multi.tests:created bucket=dmuhsu-71 2023-09-29T23:55:00.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.764 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:00.808 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:00.808 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:00.853 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-29T23:55:00.870 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-29T23:55:00.870 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.871 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.760700_581.1", 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.760700Z" 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:00.872 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-29T23:55:00.873 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-29T23:55:00.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.874 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:00.924 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:00.924 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:00.954 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:00.954+0000 7f624694b880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.955 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:00.955+0000 7f624694b880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:00.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-29T23:55:00.968 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-29T23:55:00.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-29T23:55:00.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-29T23:55:00.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:00.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-29T23:55:00.970 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.760700_581.1", 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-29T23:55:00.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-29T23:55:00.974 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-29T23:55:00Z" 2023-09-29T23:55:00.975 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-29T23:55:00.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031700.760700_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.760700Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:55:00Z" } 2023-09-29T23:55:00.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.760700_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.760700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:55:00.976 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:00.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-29T23:55:00.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-29T23:55:00.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:00.977 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:01.022 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:01.022 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:01.055 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:01.055+0000 7f3dce11a880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:01.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:01.056+0000 7f3dce11a880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:01.070 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:55:01.070 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:55:01.070 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:55:01.070 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:01.071 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:01.072 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:01.073 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.322172_577.1", 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:01.074 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.322172Z", 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:01.075 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:01.076 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:55:01Z" 2023-09-29T23:55:01.077 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:55:01.078 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031700.322172_577.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.322172Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:55:01Z" } 2023-09-29T23:55:01.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.322172_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.322172Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:55:01.078 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:01.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:01.079 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696031700.760700_581.1 target=1_1696031700.322172_577.1 2023-09-29T23:55:06.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:06.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:55:06.080 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:55:06.133 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:06.133 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:06.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:06.172+0000 7f468fe0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:06.173 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:06.173+0000 7f468fe0a880 20 RGW-SYNC:meta: read sync status 2023-09-29T23:55:06.188 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-29T23:55:06.188 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-29T23:55:06.188 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:06.189 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-29T23:55:06.190 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-29T23:55:06.191 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-29T23:55:06.192 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.760700_581.1", 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:06.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-29T23:55:06.194 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.195 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-29T23:55:06.196 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-29T23:55:06.196 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-29T23:55:06.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-29T23:55:06.196 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-29T23:55:06Z" 2023-09-29T23:55:06.196 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-29T23:55:06.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696031700.760700_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.760700Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:55:06Z" } 2023-09-29T23:55:06.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.760700_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.760700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-29T23:55:06.198 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:06.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-29T23:55:06.198 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-29T23:55:06.198 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-29T23:55:06.198 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:55:06.199 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T23:55:06.199+0000 7f2ca8046640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 115458) UID: 0 2023-09-29T23:55:06.200 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-29T23:55:06.199+0000 7f2cab897880 -1 shutting down 2023-09-29T23:55:06.290 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-29T23:55:06.291 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:06.291 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:06.291 DEBUG:boto:Method: HEAD 2023-09-29T23:55:06.291 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:06.291 DEBUG:boto:Data: 2023-09-29T23:55:06.291 DEBUG:boto:Headers: {} 2023-09-29T23:55:06.291 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.292 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.292 DEBUG:boto:Params: {} 2023-09-29T23:55:06.292 DEBUG:boto:Token: None 2023-09-29T23:55:06.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/ 2023-09-29T23:55:06.292 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-29T23:55:06.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-29T23:55:06.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054b1adb8e7433ff2-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.296 DEBUG:boto:path=/dmuhsu-71/obj11 2023-09-29T23:55:06.296 DEBUG:boto:auth_path=/dmuhsu-71/obj11 2023-09-29T23:55:06.296 DEBUG:boto:Method: PUT 2023-09-29T23:55:06.296 DEBUG:boto:Path: /dmuhsu-71/obj11 2023-09-29T23:55:06.296 DEBUG:boto:Data: 2023-09-29T23:55:06.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:55:06.296 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.296 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.297 DEBUG:boto:Params: {} 2023-09-29T23:55:06.297 DEBUG:boto:Token: None 2023-09-29T23:55:06.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/obj11 2023-09-29T23:55:06.297 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+Klo3VSZI3aQWRFejn9gl4DSX2k= 2023-09-29T23:55:06.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Klo3VSZI3aQWRFejn9gl4DSX2k='} 2023-09-29T23:55:06.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a29e46b0001319a-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.304 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:06.304 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:06.304 DEBUG:boto:Method: HEAD 2023-09-29T23:55:06.304 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:06.304 DEBUG:boto:Data: 2023-09-29T23:55:06.304 DEBUG:boto:Headers: {} 2023-09-29T23:55:06.304 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.305 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.305 DEBUG:boto:Params: {} 2023-09-29T23:55:06.305 DEBUG:boto:Token: None 2023-09-29T23:55:06.305 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/ 2023-09-29T23:55:06.305 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-29T23:55:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-29T23:55:06.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e052734d08d9737d-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.307 DEBUG:boto:path=/dmuhsu-71/obj22 2023-09-29T23:55:06.307 DEBUG:boto:auth_path=/dmuhsu-71/obj22 2023-09-29T23:55:06.307 DEBUG:boto:Method: PUT 2023-09-29T23:55:06.308 DEBUG:boto:Path: /dmuhsu-71/obj22 2023-09-29T23:55:06.308 DEBUG:boto:Data: 2023-09-29T23:55:06.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:55:06.308 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.308 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.308 DEBUG:boto:Params: {} 2023-09-29T23:55:06.308 DEBUG:boto:Token: None 2023-09-29T23:55:06.308 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/obj22 2023-09-29T23:55:06.309 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SSBojpGxdwTui+nE6ru443Qq6os= 2023-09-29T23:55:06.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SSBojpGxdwTui+nE6ru443Qq6os='} 2023-09-29T23:55:06.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003efde1059f89e1a5-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.355 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:06.355 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:06.356 DEBUG:boto:Method: HEAD 2023-09-29T23:55:06.356 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:06.356 DEBUG:boto:Data: 2023-09-29T23:55:06.356 DEBUG:boto:Headers: {} 2023-09-29T23:55:06.356 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.356 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.356 DEBUG:boto:Params: {} 2023-09-29T23:55:06.356 DEBUG:boto:Token: None 2023-09-29T23:55:06.357 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/ 2023-09-29T23:55:06.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-29T23:55:06.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-29T23:55:06.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceb80da470328394-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.359 DEBUG:boto:path=/dmuhsu-71/obj33 2023-09-29T23:55:06.359 DEBUG:boto:auth_path=/dmuhsu-71/obj33 2023-09-29T23:55:06.359 DEBUG:boto:Method: PUT 2023-09-29T23:55:06.359 DEBUG:boto:Path: /dmuhsu-71/obj33 2023-09-29T23:55:06.359 DEBUG:boto:Data: 2023-09-29T23:55:06.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:55:06.360 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.360 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.360 DEBUG:boto:Params: {} 2023-09-29T23:55:06.360 DEBUG:boto:Token: None 2023-09-29T23:55:06.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/obj33 2023-09-29T23:55:06.360 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UIORxCWZ4sBlypUF9TYMld4Uung= 2023-09-29T23:55:06.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UIORxCWZ4sBlypUF9TYMld4Uung='} 2023-09-29T23:55:06.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e5269364fa99d98-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.365 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:06.365 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:06.366 DEBUG:boto:Method: HEAD 2023-09-29T23:55:06.366 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:06.366 DEBUG:boto:Data: 2023-09-29T23:55:06.366 DEBUG:boto:Headers: {} 2023-09-29T23:55:06.366 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.366 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.366 DEBUG:boto:Params: {} 2023-09-29T23:55:06.366 DEBUG:boto:Token: None 2023-09-29T23:55:06.367 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/ 2023-09-29T23:55:06.367 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-29T23:55:06.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-29T23:55:06.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c35e6e63cbd10b7-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.369 DEBUG:boto:path=/dmuhsu-71/obj44 2023-09-29T23:55:06.369 DEBUG:boto:auth_path=/dmuhsu-71/obj44 2023-09-29T23:55:06.369 DEBUG:boto:Method: PUT 2023-09-29T23:55:06.369 DEBUG:boto:Path: /dmuhsu-71/obj44 2023-09-29T23:55:06.370 DEBUG:boto:Data: 2023-09-29T23:55:06.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:55:06.370 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.370 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.370 DEBUG:boto:Params: {} 2023-09-29T23:55:06.370 DEBUG:boto:Token: None 2023-09-29T23:55:06.370 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/obj44 2023-09-29T23:55:06.371 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NY7IzWnfmgCqkdP2Vxce3/BbbDs= 2023-09-29T23:55:06.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NY7IzWnfmgCqkdP2Vxce3/BbbDs='} 2023-09-29T23:55:06.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a530a6d5a57cb46-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.415 DEBUG:boto:path=/dmuhsu-71/ 2023-09-29T23:55:06.415 DEBUG:boto:auth_path=/dmuhsu-71/ 2023-09-29T23:55:06.415 DEBUG:boto:Method: HEAD 2023-09-29T23:55:06.415 DEBUG:boto:Path: /dmuhsu-71/ 2023-09-29T23:55:06.415 DEBUG:boto:Data: 2023-09-29T23:55:06.415 DEBUG:boto:Headers: {} 2023-09-29T23:55:06.415 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.415 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.415 DEBUG:boto:Params: {} 2023-09-29T23:55:06.416 DEBUG:boto:Token: None 2023-09-29T23:55:06.416 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/ 2023-09-29T23:55:06.416 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-29T23:55:06.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-29T23:55:06.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052d885013ea79282-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.418 DEBUG:boto:path=/dmuhsu-71/obj55 2023-09-29T23:55:06.418 DEBUG:boto:auth_path=/dmuhsu-71/obj55 2023-09-29T23:55:06.419 DEBUG:boto:Method: PUT 2023-09-29T23:55:06.419 DEBUG:boto:Path: /dmuhsu-71/obj55 2023-09-29T23:55:06.419 DEBUG:boto:Data: 2023-09-29T23:55:06.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:55:06.419 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-29T23:55:06.419 DEBUG:boto:Port: 8000 2023-09-29T23:55:06.419 DEBUG:boto:Params: {} 2023-09-29T23:55:06.419 DEBUG:boto:Token: None 2023-09-29T23:55:06.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:55:06 GMT /dmuhsu-71/obj55 2023-09-29T23:55:06.420 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5KZgMrR/HyPddCgIG8DpEoEXUZI= 2023-09-29T23:55:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5KZgMrR/HyPddCgIG8DpEoEXUZI='} 2023-09-29T23:55:06.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f9912e5f6ec9123-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:55:06.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:55:06.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:55:06.424 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T23:55:06.477 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:06.477 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:06.510 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:06.510+0000 7f8829811880 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T23:55:06.511 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-29T23:55:06.511+0000 7f8829811880 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T23:55:06.534 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T23:55:06.534 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T23:55:06.536 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T23:55:06.537 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T23:55:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T23:55:06.555 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:55:06.555 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:55:06.555 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:55:06.555 INFO:teuthology.orchestra.run.smithi184.stderr:curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T23:55:07.556 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-29T23:55:07.577 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:55:07.577 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:55:07.578 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T23:55:07.579 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-29T23:55:07.579 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-71 2023-09-29T23:55:07.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:07.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:07.580 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-29T23:55:07.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:07.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:07.703 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-71 2023-09-29T23:55:07.717 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-71 2023-09-29T23:55:07.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:07.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:07.718 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:55:07.773 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:07.773 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:07.832 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:07.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-29T23:55:12.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:12.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-29T23:55:17.833 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:17.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-29T23:55:22.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:22.834+0000 7fbaadaba880 1 bucket sync caught up with source: 2023-09-29T23:55:22.835 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.36.6, 00000000002.49.6, 00000000002.52.6] 2023-09-29T23:55:22.835 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.36.6, 00000000002.49.6, 00000000002.52.6] 2023-09-29T23:55:22.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:22.834+0000 7fbaadaba880 0 bucket checkpoint complete 2023-09-29T23:55:22.854 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-71 2023-09-29T23:55:22.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:22.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:55:22.855 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:55:22.911 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:55:22.911 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:55:22.972 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:22.971+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:27.973 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:27.972+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:32.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:32.973+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:37.974 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:37.974+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:42.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:42.974+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:47.975 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:47.975+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:52.976 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:52.976+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:55:57.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:55:57.976+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:02.977 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:02.977+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:07.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:07.977+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:12.978 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:12.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:17.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:17.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:22.979 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:22.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:27.980 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:27.979+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:32.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:32.980+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:37.981 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:37.981+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:42.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:42.981+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:47.982 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:47.982+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:52.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:52.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:56:57.983 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:56:57.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:02.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:02.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:07.984 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:07.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:12.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:12.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:17.985 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:17.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:22.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:22.985+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:27.986 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:27.986+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:32.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:32.987+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:37.987 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:37.987+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:42.988 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:42.988+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:47.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:47.989+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:52.989 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:52.989+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:57:57.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:57:57.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:02.990 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:02.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:07.991 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:07.991+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:12.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:12.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:17.992 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:17.991+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:22.993 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:22.992+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:27.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:27.993+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:32.994 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:32.994+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:37.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:37.994+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:42.995 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:42.995+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:47.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:47.995+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:52.996 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:52.996+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:58:57.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:58:57.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:02.997 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:02.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:07.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:07.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:12.998 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:12.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:17.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:17.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:22.999 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:22.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:28.000 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:28.000+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:33.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:33.000+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:38.001 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:38.001+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:43.002 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:43.001+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:48.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:48.002+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:53.003 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:53.003+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-29T23:59:58.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-29T23:59:58.003+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:00:03.004 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:03.004+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:00:08.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:08.005+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:00:13.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:13.004+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:00:18.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:18.005+0000 7f8f16f20880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:00:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:18.005+0000 7f8f16f20880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:00:18.007 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:18.005+0000 7f8f16f20880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:00:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-30T00:00:18.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2023-09-30T00:00:18.033 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:18.033 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:18.033 DEBUG:boto:Method: PUT 2023-09-30T00:00:18.033 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:18.033 DEBUG:boto:Data: 2023-09-30T00:00:18.033 DEBUG:boto:Headers: {} 2023-09-30T00:00:18.033 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:18.034 DEBUG:boto:Port: 8000 2023-09-30T00:00:18.034 DEBUG:boto:Params: {} 2023-09-30T00:00:18.034 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:00:18.034 DEBUG:boto:Token: None 2023-09-30T00:00:18.034 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:00:18 GMT /dmuhsu-72/ 2023-09-30T00:00:18.035 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BnHpl12/l9P6GhXTtUoV/BQ16cA= 2023-09-30T00:00:18.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BnHpl12/l9P6GhXTtUoV/BQ16cA=', 'Content-Length': '0'} 2023-09-30T00:00:18.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000924869d3d71874df-0065176512-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:18.063 DEBUG:rgw_multi.tests:created bucket=dmuhsu-72 2023-09-30T00:00:18.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:18.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:18.063 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:00:18.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:18.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:18.162 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:00:18.182 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:00:18.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:18.182 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:00:18.060994Z" 2023-09-30T00:00:18.183 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:18.184 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:18.184 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:00:18.184 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:00:18.184 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:18.184 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:00:18.185 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:18.185 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:00:18.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:18.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:18.186 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:00:18.239 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:18.240 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:18.273 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:18.272+0000 7f7004136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:18.273 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:18.273+0000 7f7004136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:00:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:00:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:00:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:00:18.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:18.295 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:00:18Z" 2023-09-30T00:00:18.295 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:00:18.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032018.052336_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:00:18.052336Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:00:18Z" } 2023-09-30T00:00:18.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.052336_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.052336Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:00:18.296 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:18.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.052336_610.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:18.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696032018.060994_613.1 target=1_1696032018.052336_610.1 2023-09-30T00:00:23.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:23.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:23.298 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:00:23.350 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:23.351 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:23.383 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:23.382+0000 7f8efb215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:23.384 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:23.383+0000 7f8efb215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:00:23.398 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:23.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:23.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:00:23.401 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:23.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:00:23Z" 2023-09-30T00:00:23.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:00:23.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032018.052336_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:00:18.052336Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:00:23Z" } 2023-09-30T00:00:23.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.052336_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.052336Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:00:23.406 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:23.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.052336_610.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:23.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696032018.060994_613.1 target=1_1696032018.052336_610.1 2023-09-30T00:00:28.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:28.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:28.408 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:00:28.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:28.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:28.495 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:28.495+0000 7f1f53044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:28.496 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:28.496+0000 7f1f53044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.511 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:00:28.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:28.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:28.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:00:28.515 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:00:28.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:00:28.517 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:00:28.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032018.060994_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:00:18.060994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:00:28Z" } 2023-09-30T00:00:28.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.060994_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.060994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:00:28.518 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:28.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:28.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:00:28.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:00:28.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:28.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:00:28.519 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:00:28.573 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:28.573 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:28.618 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:28.618+0000 7f36e3813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:28.619 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:28.619+0000 7f36e3813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:00:28.639 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:00:28.640 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:00:28.641 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:28.642 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:28.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:00:28.644 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:00:28.645 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:00:28.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032018.060994_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:00:18.060994Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:00:28Z" } 2023-09-30T00:00:28.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.060994_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.060994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:00:28.647 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:28.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:00:28.647 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:00:28.647 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-30T00:00:28.647 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:00:28.648 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-30T00:00:28.647+0000 7f15e156d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 117059) UID: 0 2023-09-30T00:00:28.648 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-30T00:00:28.648+0000 7f15e7a10880 -1 shutting down 2023-09-30T00:00:34.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-30T00:00:34.749 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-30T00:00:34.749 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:34.749 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:34.750 DEBUG:boto:Method: HEAD 2023-09-30T00:00:34.750 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:34.750 DEBUG:boto:Data: 2023-09-30T00:00:34.750 DEBUG:boto:Headers: {} 2023-09-30T00:00:34.750 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.750 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.750 DEBUG:boto:Params: {} 2023-09-30T00:00:34.750 DEBUG:boto:Token: None 2023-09-30T00:00:34.751 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/ 2023-09-30T00:00:34.751 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:00:34.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:00:34.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bd9995ac660f80e-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.754 DEBUG:boto:path=/dmuhsu-72/obj11 2023-09-30T00:00:34.754 DEBUG:boto:auth_path=/dmuhsu-72/obj11 2023-09-30T00:00:34.754 DEBUG:boto:Method: PUT 2023-09-30T00:00:34.755 DEBUG:boto:Path: /dmuhsu-72/obj11 2023-09-30T00:00:34.755 DEBUG:boto:Data: 2023-09-30T00:00:34.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:34.755 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.755 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.755 DEBUG:boto:Params: {} 2023-09-30T00:00:34.755 DEBUG:boto:Token: None 2023-09-30T00:00:34.755 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/obj11 2023-09-30T00:00:34.755 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TjFnUhGXV7zXfL6bWtfZKKR5EP8= 2023-09-30T00:00:34.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TjFnUhGXV7zXfL6bWtfZKKR5EP8='} 2023-09-30T00:00:34.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55af2a2d6760145-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.766 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:34.766 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:34.767 DEBUG:boto:Method: HEAD 2023-09-30T00:00:34.767 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:34.767 DEBUG:boto:Data: 2023-09-30T00:00:34.767 DEBUG:boto:Headers: {} 2023-09-30T00:00:34.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.767 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.767 DEBUG:boto:Params: {} 2023-09-30T00:00:34.767 DEBUG:boto:Token: None 2023-09-30T00:00:34.768 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/ 2023-09-30T00:00:34.768 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:00:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:00:34.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faa72298fb081e77-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.770 DEBUG:boto:path=/dmuhsu-72/obj22 2023-09-30T00:00:34.771 DEBUG:boto:auth_path=/dmuhsu-72/obj22 2023-09-30T00:00:34.771 DEBUG:boto:Method: PUT 2023-09-30T00:00:34.771 DEBUG:boto:Path: /dmuhsu-72/obj22 2023-09-30T00:00:34.771 DEBUG:boto:Data: 2023-09-30T00:00:34.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:34.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.771 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.771 DEBUG:boto:Params: {} 2023-09-30T00:00:34.771 DEBUG:boto:Token: None 2023-09-30T00:00:34.772 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/obj22 2023-09-30T00:00:34.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:07XVZ4zeGM5viLltumPJa+QcuiY= 2023-09-30T00:00:34.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:07XVZ4zeGM5viLltumPJa+QcuiY='} 2023-09-30T00:00:34.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000189c68d8a6ddb49f-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.776 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:34.777 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:34.777 DEBUG:boto:Method: HEAD 2023-09-30T00:00:34.777 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:34.777 DEBUG:boto:Data: 2023-09-30T00:00:34.777 DEBUG:boto:Headers: {} 2023-09-30T00:00:34.778 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.778 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.778 DEBUG:boto:Params: {} 2023-09-30T00:00:34.778 DEBUG:boto:Token: None 2023-09-30T00:00:34.778 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/ 2023-09-30T00:00:34.778 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:00:34.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:00:34.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000404f9bbdd3b645d6-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.781 DEBUG:boto:path=/dmuhsu-72/obj33 2023-09-30T00:00:34.781 DEBUG:boto:auth_path=/dmuhsu-72/obj33 2023-09-30T00:00:34.781 DEBUG:boto:Method: PUT 2023-09-30T00:00:34.781 DEBUG:boto:Path: /dmuhsu-72/obj33 2023-09-30T00:00:34.781 DEBUG:boto:Data: 2023-09-30T00:00:34.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:34.781 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.782 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.782 DEBUG:boto:Params: {} 2023-09-30T00:00:34.782 DEBUG:boto:Token: None 2023-09-30T00:00:34.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/obj33 2023-09-30T00:00:34.782 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cKX+wY6FfL9OjKNfuguBqnRLumY= 2023-09-30T00:00:34.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cKX+wY6FfL9OjKNfuguBqnRLumY='} 2023-09-30T00:00:34.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b604a47afd9b523-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.789 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:34.789 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:34.789 DEBUG:boto:Method: HEAD 2023-09-30T00:00:34.790 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:34.790 DEBUG:boto:Data: 2023-09-30T00:00:34.790 DEBUG:boto:Headers: {} 2023-09-30T00:00:34.790 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.790 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.790 DEBUG:boto:Params: {} 2023-09-30T00:00:34.790 DEBUG:boto:Token: None 2023-09-30T00:00:34.790 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/ 2023-09-30T00:00:34.791 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:00:34.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:00:34.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a4c48568339383-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.793 DEBUG:boto:path=/dmuhsu-72/obj44 2023-09-30T00:00:34.793 DEBUG:boto:auth_path=/dmuhsu-72/obj44 2023-09-30T00:00:34.793 DEBUG:boto:Method: PUT 2023-09-30T00:00:34.793 DEBUG:boto:Path: /dmuhsu-72/obj44 2023-09-30T00:00:34.794 DEBUG:boto:Data: 2023-09-30T00:00:34.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:34.794 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.794 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.794 DEBUG:boto:Params: {} 2023-09-30T00:00:34.794 DEBUG:boto:Token: None 2023-09-30T00:00:34.794 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/obj44 2023-09-30T00:00:34.794 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NVETEW08T6IK+aiYuxQSXj9HBSE= 2023-09-30T00:00:34.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NVETEW08T6IK+aiYuxQSXj9HBSE='} 2023-09-30T00:00:34.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084a9585a012beba2-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.843 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:34.843 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:34.843 DEBUG:boto:Method: HEAD 2023-09-30T00:00:34.843 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:34.843 DEBUG:boto:Data: 2023-09-30T00:00:34.843 DEBUG:boto:Headers: {} 2023-09-30T00:00:34.844 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.844 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.844 DEBUG:boto:Params: {} 2023-09-30T00:00:34.844 DEBUG:boto:Token: None 2023-09-30T00:00:34.844 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/ 2023-09-30T00:00:34.844 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:00:34.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:00:34.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000740466b349d549a2-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.847 DEBUG:boto:path=/dmuhsu-72/obj55 2023-09-30T00:00:34.847 DEBUG:boto:auth_path=/dmuhsu-72/obj55 2023-09-30T00:00:34.847 DEBUG:boto:Method: PUT 2023-09-30T00:00:34.847 DEBUG:boto:Path: /dmuhsu-72/obj55 2023-09-30T00:00:34.847 DEBUG:boto:Data: 2023-09-30T00:00:34.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:34.847 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:34.847 DEBUG:boto:Port: 8000 2023-09-30T00:00:34.847 DEBUG:boto:Params: {} 2023-09-30T00:00:34.848 DEBUG:boto:Token: None 2023-09-30T00:00:34.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:34 GMT /dmuhsu-72/obj55 2023-09-30T00:00:34.848 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zqi1NPMGyJJLiilR6b4sE+SryCs= 2023-09-30T00:00:34.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqi1NPMGyJJLiilR6b4sE+SryCs='} 2023-09-30T00:00:34.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eecc0336eca1d9fa-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:34.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:34.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:34.853 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:34.906 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:34.906 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:34.958 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:34.958 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:34.961 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 5 2023-09-30T00:00:35.012 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.013 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.013 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.013 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.013 DEBUG:boto:Data: 2023-09-30T00:00:35.013 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.013 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.013 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.013 DEBUG:boto:Params: {} 2023-09-30T00:00:35.014 DEBUG:boto:Token: None 2023-09-30T00:00:35.014 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.014 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b03cd72313afdb14-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.017 DEBUG:boto:path=/dmuhsu-72/obj17 2023-09-30T00:00:35.017 DEBUG:boto:auth_path=/dmuhsu-72/obj17 2023-09-30T00:00:35.017 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.017 DEBUG:boto:Path: /dmuhsu-72/obj17 2023-09-30T00:00:35.018 DEBUG:boto:Data: 2023-09-30T00:00:35.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.018 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.018 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.018 DEBUG:boto:Params: {} 2023-09-30T00:00:35.018 DEBUG:boto:Token: None 2023-09-30T00:00:35.018 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj17 2023-09-30T00:00:35.018 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3hOr8Ni86kZjUHLjCf4MOqLBpdM= 2023-09-30T00:00:35.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3hOr8Ni86kZjUHLjCf4MOqLBpdM='} 2023-09-30T00:00:35.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a66aef88d3809a6-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.028 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.028 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.028 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.028 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.028 DEBUG:boto:Data: 2023-09-30T00:00:35.029 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.029 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.029 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.029 DEBUG:boto:Params: {} 2023-09-30T00:00:35.029 DEBUG:boto:Token: None 2023-09-30T00:00:35.029 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.029 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003acd69cc1c84fa3a-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.032 DEBUG:boto:path=/dmuhsu-72/obj34 2023-09-30T00:00:35.032 DEBUG:boto:auth_path=/dmuhsu-72/obj34 2023-09-30T00:00:35.032 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.032 DEBUG:boto:Path: /dmuhsu-72/obj34 2023-09-30T00:00:35.032 DEBUG:boto:Data: 2023-09-30T00:00:35.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.032 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.032 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.032 DEBUG:boto:Params: {} 2023-09-30T00:00:35.033 DEBUG:boto:Token: None 2023-09-30T00:00:35.033 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj34 2023-09-30T00:00:35.033 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Sq1Pnvn8EQDWWR5RlIa122ogpzU= 2023-09-30T00:00:35.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Sq1Pnvn8EQDWWR5RlIa122ogpzU='} 2023-09-30T00:00:35.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064d966516634b619-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.079 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.079 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.079 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.079 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.079 DEBUG:boto:Data: 2023-09-30T00:00:35.079 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.079 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.080 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.080 DEBUG:boto:Params: {} 2023-09-30T00:00:35.080 DEBUG:boto:Token: None 2023-09-30T00:00:35.080 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.080 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e978061ce7d65cff-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.083 DEBUG:boto:path=/dmuhsu-72/obj51 2023-09-30T00:00:35.083 DEBUG:boto:auth_path=/dmuhsu-72/obj51 2023-09-30T00:00:35.083 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.083 DEBUG:boto:Path: /dmuhsu-72/obj51 2023-09-30T00:00:35.084 DEBUG:boto:Data: 2023-09-30T00:00:35.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.084 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.084 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.084 DEBUG:boto:Params: {} 2023-09-30T00:00:35.084 DEBUG:boto:Token: None 2023-09-30T00:00:35.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj51 2023-09-30T00:00:35.084 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ScG0x3l7NuNw+3/9+mSosMdeh6Y= 2023-09-30T00:00:35.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ScG0x3l7NuNw+3/9+mSosMdeh6Y='} 2023-09-30T00:00:35.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000901e2d7e1a02ffca-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.094 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.094 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.094 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.094 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.094 DEBUG:boto:Data: 2023-09-30T00:00:35.094 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.094 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.095 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.095 DEBUG:boto:Params: {} 2023-09-30T00:00:35.095 DEBUG:boto:Token: None 2023-09-30T00:00:35.095 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.095 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ffc97d0a0b44849-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.098 DEBUG:boto:path=/dmuhsu-72/obj68 2023-09-30T00:00:35.098 DEBUG:boto:auth_path=/dmuhsu-72/obj68 2023-09-30T00:00:35.098 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.098 DEBUG:boto:Path: /dmuhsu-72/obj68 2023-09-30T00:00:35.098 DEBUG:boto:Data: 2023-09-30T00:00:35.098 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.098 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.098 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.099 DEBUG:boto:Params: {} 2023-09-30T00:00:35.099 DEBUG:boto:Token: None 2023-09-30T00:00:35.099 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj68 2023-09-30T00:00:35.099 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kNsTO/xZA+K7Lw5hRET7tUqkVDM= 2023-09-30T00:00:35.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNsTO/xZA+K7Lw5hRET7tUqkVDM='} 2023-09-30T00:00:35.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b395d82f96fd0e9-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.143 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.143 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.143 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.143 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.143 DEBUG:boto:Data: 2023-09-30T00:00:35.143 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.144 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.144 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.145 DEBUG:boto:Params: {} 2023-09-30T00:00:35.145 DEBUG:boto:Token: None 2023-09-30T00:00:35.145 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.146 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c824cd78afe3456-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.148 DEBUG:boto:path=/dmuhsu-72/obj85 2023-09-30T00:00:35.149 DEBUG:boto:auth_path=/dmuhsu-72/obj85 2023-09-30T00:00:35.149 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.149 DEBUG:boto:Path: /dmuhsu-72/obj85 2023-09-30T00:00:35.149 DEBUG:boto:Data: 2023-09-30T00:00:35.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.149 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.149 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.149 DEBUG:boto:Params: {} 2023-09-30T00:00:35.149 DEBUG:boto:Token: None 2023-09-30T00:00:35.150 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj85 2023-09-30T00:00:35.150 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:llBx9cQsEDoIz9qn3Pmo3pQ097g= 2023-09-30T00:00:35.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:llBx9cQsEDoIz9qn3Pmo3pQ097g='} 2023-09-30T00:00:35.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095fa153ad329b96f-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.162 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:35.211 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:35.273 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:35.273 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:35.278 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 10 2023-09-30T00:00:35.328 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.328 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.328 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.328 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.328 DEBUG:boto:Data: 2023-09-30T00:00:35.328 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.328 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.328 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.328 DEBUG:boto:Params: {} 2023-09-30T00:00:35.329 DEBUG:boto:Token: None 2023-09-30T00:00:35.329 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.329 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056d0c6436dcf39a8-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.332 DEBUG:boto:path=/dmuhsu-72/obj19 2023-09-30T00:00:35.332 DEBUG:boto:auth_path=/dmuhsu-72/obj19 2023-09-30T00:00:35.332 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.332 DEBUG:boto:Path: /dmuhsu-72/obj19 2023-09-30T00:00:35.332 DEBUG:boto:Data: 2023-09-30T00:00:35.333 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.333 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.333 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.333 DEBUG:boto:Params: {} 2023-09-30T00:00:35.333 DEBUG:boto:Token: None 2023-09-30T00:00:35.333 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj19 2023-09-30T00:00:35.333 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:PPZQAotqaJiKhMVDkJWEW4Ex1BA= 2023-09-30T00:00:35.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PPZQAotqaJiKhMVDkJWEW4Ex1BA='} 2023-09-30T00:00:35.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5f0ff27ca3eb5e9-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.383 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.383 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.383 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.383 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.383 DEBUG:boto:Data: 2023-09-30T00:00:35.383 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.383 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.384 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.384 DEBUG:boto:Params: {} 2023-09-30T00:00:35.384 DEBUG:boto:Token: None 2023-09-30T00:00:35.384 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.384 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7bb2dad647ecce5-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.387 DEBUG:boto:path=/dmuhsu-72/obj38 2023-09-30T00:00:35.387 DEBUG:boto:auth_path=/dmuhsu-72/obj38 2023-09-30T00:00:35.387 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.387 DEBUG:boto:Path: /dmuhsu-72/obj38 2023-09-30T00:00:35.387 DEBUG:boto:Data: 2023-09-30T00:00:35.387 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.388 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.388 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.388 DEBUG:boto:Params: {} 2023-09-30T00:00:35.388 DEBUG:boto:Token: None 2023-09-30T00:00:35.388 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj38 2023-09-30T00:00:35.388 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6FGI6IMyRt2xJi4EUlx5xuGveQE= 2023-09-30T00:00:35.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6FGI6IMyRt2xJi4EUlx5xuGveQE='} 2023-09-30T00:00:35.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af698ff4393039a5-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.392 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.392 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.392 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.392 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.392 DEBUG:boto:Data: 2023-09-30T00:00:35.392 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.392 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.392 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.392 DEBUG:boto:Params: {} 2023-09-30T00:00:35.392 DEBUG:boto:Token: None 2023-09-30T00:00:35.393 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.393 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057c569aee4267a36-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.395 DEBUG:boto:path=/dmuhsu-72/obj57 2023-09-30T00:00:35.395 DEBUG:boto:auth_path=/dmuhsu-72/obj57 2023-09-30T00:00:35.395 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.395 DEBUG:boto:Path: /dmuhsu-72/obj57 2023-09-30T00:00:35.396 DEBUG:boto:Data: 2023-09-30T00:00:35.396 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.396 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.396 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.396 DEBUG:boto:Params: {} 2023-09-30T00:00:35.396 DEBUG:boto:Token: None 2023-09-30T00:00:35.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj57 2023-09-30T00:00:35.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LN6kc/YCUcuup3D7i2F2UYvS4II= 2023-09-30T00:00:35.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LN6kc/YCUcuup3D7i2F2UYvS4II='} 2023-09-30T00:00:35.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8872d03e31fcfce-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.439 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.439 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.439 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.439 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.439 DEBUG:boto:Data: 2023-09-30T00:00:35.439 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.439 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.439 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.439 DEBUG:boto:Params: {} 2023-09-30T00:00:35.440 DEBUG:boto:Token: None 2023-09-30T00:00:35.440 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.440 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bd32db31b6e6e94-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.443 DEBUG:boto:path=/dmuhsu-72/obj76 2023-09-30T00:00:35.443 DEBUG:boto:auth_path=/dmuhsu-72/obj76 2023-09-30T00:00:35.443 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.443 DEBUG:boto:Path: /dmuhsu-72/obj76 2023-09-30T00:00:35.444 DEBUG:boto:Data: 2023-09-30T00:00:35.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.444 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.444 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.444 DEBUG:boto:Params: {} 2023-09-30T00:00:35.444 DEBUG:boto:Token: None 2023-09-30T00:00:35.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj76 2023-09-30T00:00:35.444 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z7U3mF1vLWWYQBbd65eKZ9mCZoQ= 2023-09-30T00:00:35.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z7U3mF1vLWWYQBbd65eKZ9mCZoQ='} 2023-09-30T00:00:35.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003baacc54a5ef5012-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.452 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.452 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.452 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.452 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.452 DEBUG:boto:Data: 2023-09-30T00:00:35.452 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.452 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.452 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.452 DEBUG:boto:Params: {} 2023-09-30T00:00:35.452 DEBUG:boto:Token: None 2023-09-30T00:00:35.453 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.453 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098af0eaab71e2696-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.455 DEBUG:boto:path=/dmuhsu-72/obj95 2023-09-30T00:00:35.456 DEBUG:boto:auth_path=/dmuhsu-72/obj95 2023-09-30T00:00:35.456 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.456 DEBUG:boto:Path: /dmuhsu-72/obj95 2023-09-30T00:00:35.456 DEBUG:boto:Data: 2023-09-30T00:00:35.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.456 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.456 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.456 DEBUG:boto:Params: {} 2023-09-30T00:00:35.456 DEBUG:boto:Token: None 2023-09-30T00:00:35.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj95 2023-09-30T00:00:35.457 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NZAiQ+szwSxWjZr9rk54TQnIHpk= 2023-09-30T00:00:35.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NZAiQ+szwSxWjZr9rk54TQnIHpk='} 2023-09-30T00:00:35.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042b757c39ae9b40a-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.499 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:35.547 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:35.547 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:35.624 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:35.624 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:35.630 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 15 2023-09-30T00:00:35.686 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.686 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.686 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.686 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.686 DEBUG:boto:Data: 2023-09-30T00:00:35.686 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.686 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.687 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.687 DEBUG:boto:Params: {} 2023-09-30T00:00:35.687 DEBUG:boto:Token: None 2023-09-30T00:00:35.687 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f84019c38fba4dc9-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.690 DEBUG:boto:path=/dmuhsu-72/obj23 2023-09-30T00:00:35.690 DEBUG:boto:auth_path=/dmuhsu-72/obj23 2023-09-30T00:00:35.690 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.690 DEBUG:boto:Path: /dmuhsu-72/obj23 2023-09-30T00:00:35.691 DEBUG:boto:Data: 2023-09-30T00:00:35.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.691 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.691 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.691 DEBUG:boto:Params: {} 2023-09-30T00:00:35.691 DEBUG:boto:Token: None 2023-09-30T00:00:35.691 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj23 2023-09-30T00:00:35.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JgzDtm1670SWMsERD48qJyc7PiA= 2023-09-30T00:00:35.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JgzDtm1670SWMsERD48qJyc7PiA='} 2023-09-30T00:00:35.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000273c9ad95ac27944-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.696 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.697 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.697 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.697 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.697 DEBUG:boto:Data: 2023-09-30T00:00:35.697 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.697 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.697 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.697 DEBUG:boto:Params: {} 2023-09-30T00:00:35.697 DEBUG:boto:Token: None 2023-09-30T00:00:35.697 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.698 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d97fcb2e43ecf687-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.700 DEBUG:boto:path=/dmuhsu-72/obj46 2023-09-30T00:00:35.700 DEBUG:boto:auth_path=/dmuhsu-72/obj46 2023-09-30T00:00:35.700 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.700 DEBUG:boto:Path: /dmuhsu-72/obj46 2023-09-30T00:00:35.700 DEBUG:boto:Data: 2023-09-30T00:00:35.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.701 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.701 DEBUG:boto:Params: {} 2023-09-30T00:00:35.701 DEBUG:boto:Token: None 2023-09-30T00:00:35.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj46 2023-09-30T00:00:35.701 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E265vdwud4+zqObD5Z8RLz5/fE8= 2023-09-30T00:00:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E265vdwud4+zqObD5Z8RLz5/fE8='} 2023-09-30T00:00:35.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7b840a5445ade83-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.706 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.706 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.707 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.707 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.707 DEBUG:boto:Data: 2023-09-30T00:00:35.707 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.707 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.707 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.707 DEBUG:boto:Params: {} 2023-09-30T00:00:35.707 DEBUG:boto:Token: None 2023-09-30T00:00:35.707 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.707 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfec5906ebe32313-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.710 DEBUG:boto:path=/dmuhsu-72/obj69 2023-09-30T00:00:35.710 DEBUG:boto:auth_path=/dmuhsu-72/obj69 2023-09-30T00:00:35.710 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.711 DEBUG:boto:Path: /dmuhsu-72/obj69 2023-09-30T00:00:35.711 DEBUG:boto:Data: 2023-09-30T00:00:35.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.711 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.711 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.711 DEBUG:boto:Params: {} 2023-09-30T00:00:35.711 DEBUG:boto:Token: None 2023-09-30T00:00:35.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj69 2023-09-30T00:00:35.711 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:A5GW9/C4v654Ny1elANfjTpARXo= 2023-09-30T00:00:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5GW9/C4v654Ny1elANfjTpARXo='} 2023-09-30T00:00:35.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d1aacdad02aca47-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.762 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.762 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.762 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.762 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.762 DEBUG:boto:Data: 2023-09-30T00:00:35.762 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.762 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.762 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.762 DEBUG:boto:Params: {} 2023-09-30T00:00:35.763 DEBUG:boto:Token: None 2023-09-30T00:00:35.763 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.763 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030817e7dc929bf79-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.766 DEBUG:boto:path=/dmuhsu-72/obj92 2023-09-30T00:00:35.766 DEBUG:boto:auth_path=/dmuhsu-72/obj92 2023-09-30T00:00:35.766 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.766 DEBUG:boto:Path: /dmuhsu-72/obj92 2023-09-30T00:00:35.766 DEBUG:boto:Data: 2023-09-30T00:00:35.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.767 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.767 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.767 DEBUG:boto:Params: {} 2023-09-30T00:00:35.767 DEBUG:boto:Token: None 2023-09-30T00:00:35.767 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj92 2023-09-30T00:00:35.767 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F4+gW26YtGpH4sj9vH49QMQzKn8= 2023-09-30T00:00:35.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F4+gW26YtGpH4sj9vH49QMQzKn8='} 2023-09-30T00:00:35.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073ffd653f2a2af82-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.773 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.773 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.773 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.773 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.773 DEBUG:boto:Data: 2023-09-30T00:00:35.773 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.773 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.773 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.773 DEBUG:boto:Params: {} 2023-09-30T00:00:35.774 DEBUG:boto:Token: None 2023-09-30T00:00:35.774 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.774 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014d0272b0836aa6d-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.777 DEBUG:boto:path=/dmuhsu-72/obj115 2023-09-30T00:00:35.777 DEBUG:boto:auth_path=/dmuhsu-72/obj115 2023-09-30T00:00:35.777 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.777 DEBUG:boto:Path: /dmuhsu-72/obj115 2023-09-30T00:00:35.777 DEBUG:boto:Data: 2023-09-30T00:00:35.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.777 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.777 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.777 DEBUG:boto:Params: {} 2023-09-30T00:00:35.778 DEBUG:boto:Token: None 2023-09-30T00:00:35.778 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj115 2023-09-30T00:00:35.778 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RtXkO20fJo45PJ6LU8ID9CSpKZY= 2023-09-30T00:00:35.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RtXkO20fJo45PJ6LU8ID9CSpKZY='} 2023-09-30T00:00:35.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d3ca37efb32a5f1-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:35.785 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:35.834 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:35.834 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:35.920 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 20 2023-09-30T00:00:35.993 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:35.993 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:35.993 DEBUG:boto:Method: HEAD 2023-09-30T00:00:35.993 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:35.994 DEBUG:boto:Data: 2023-09-30T00:00:35.994 DEBUG:boto:Headers: {} 2023-09-30T00:00:35.994 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.994 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.994 DEBUG:boto:Params: {} 2023-09-30T00:00:35.994 DEBUG:boto:Token: None 2023-09-30T00:00:35.994 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/ 2023-09-30T00:00:35.995 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:00:35.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:00:35.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab633dbe534b118e-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:35.998 DEBUG:boto:path=/dmuhsu-72/obj29 2023-09-30T00:00:35.998 DEBUG:boto:auth_path=/dmuhsu-72/obj29 2023-09-30T00:00:35.999 DEBUG:boto:Method: PUT 2023-09-30T00:00:35.999 DEBUG:boto:Path: /dmuhsu-72/obj29 2023-09-30T00:00:35.999 DEBUG:boto:Data: 2023-09-30T00:00:35.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:35.999 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:35.999 DEBUG:boto:Port: 8000 2023-09-30T00:00:35.999 DEBUG:boto:Params: {} 2023-09-30T00:00:35.999 DEBUG:boto:Token: None 2023-09-30T00:00:35.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:35 GMT /dmuhsu-72/obj29 2023-09-30T00:00:36.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R/oYbwcqbX5nYX8xoIPOSTbN1p4= 2023-09-30T00:00:36.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R/oYbwcqbX5nYX8xoIPOSTbN1p4='} 2023-09-30T00:00:36.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004984ce35b5ea4df8-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.005 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.005 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.005 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.005 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.005 DEBUG:boto:Data: 2023-09-30T00:00:36.006 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.006 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.006 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.006 DEBUG:boto:Params: {} 2023-09-30T00:00:36.006 DEBUG:boto:Token: None 2023-09-30T00:00:36.006 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.006 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000409f28769fb79ea2-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.009 DEBUG:boto:path=/dmuhsu-72/obj58 2023-09-30T00:00:36.010 DEBUG:boto:auth_path=/dmuhsu-72/obj58 2023-09-30T00:00:36.010 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.010 DEBUG:boto:Path: /dmuhsu-72/obj58 2023-09-30T00:00:36.010 DEBUG:boto:Data: 2023-09-30T00:00:36.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.010 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.010 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.010 DEBUG:boto:Params: {} 2023-09-30T00:00:36.011 DEBUG:boto:Token: None 2023-09-30T00:00:36.011 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj58 2023-09-30T00:00:36.011 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qRdrm16gIFL0dkpvW8vrJkmEuto= 2023-09-30T00:00:36.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qRdrm16gIFL0dkpvW8vrJkmEuto='} 2023-09-30T00:00:36.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd3693c95f6a4824-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.059 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.059 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.059 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.059 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.059 DEBUG:boto:Data: 2023-09-30T00:00:36.060 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.060 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.060 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.060 DEBUG:boto:Params: {} 2023-09-30T00:00:36.060 DEBUG:boto:Token: None 2023-09-30T00:00:36.060 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.060 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002120fe4975a98160-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.063 DEBUG:boto:path=/dmuhsu-72/obj87 2023-09-30T00:00:36.063 DEBUG:boto:auth_path=/dmuhsu-72/obj87 2023-09-30T00:00:36.063 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.064 DEBUG:boto:Path: /dmuhsu-72/obj87 2023-09-30T00:00:36.064 DEBUG:boto:Data: 2023-09-30T00:00:36.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.064 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.064 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.064 DEBUG:boto:Params: {} 2023-09-30T00:00:36.064 DEBUG:boto:Token: None 2023-09-30T00:00:36.064 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj87 2023-09-30T00:00:36.064 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oc2vABim/HkvkJ7tH1yboPJrtXI= 2023-09-30T00:00:36.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oc2vABim/HkvkJ7tH1yboPJrtXI='} 2023-09-30T00:00:36.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000126cc538fc5d8206-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.069 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.069 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.070 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.070 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.070 DEBUG:boto:Data: 2023-09-30T00:00:36.070 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.070 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.070 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.070 DEBUG:boto:Params: {} 2023-09-30T00:00:36.070 DEBUG:boto:Token: None 2023-09-30T00:00:36.070 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.071 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bbf225e73d6fafc-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.073 DEBUG:boto:path=/dmuhsu-72/obj116 2023-09-30T00:00:36.074 DEBUG:boto:auth_path=/dmuhsu-72/obj116 2023-09-30T00:00:36.074 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.074 DEBUG:boto:Path: /dmuhsu-72/obj116 2023-09-30T00:00:36.074 DEBUG:boto:Data: 2023-09-30T00:00:36.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.074 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.074 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.074 DEBUG:boto:Params: {} 2023-09-30T00:00:36.074 DEBUG:boto:Token: None 2023-09-30T00:00:36.075 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj116 2023-09-30T00:00:36.075 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VIX68o4ioYpi8FqaSbxTgi+ojcQ= 2023-09-30T00:00:36.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VIX68o4ioYpi8FqaSbxTgi+ojcQ='} 2023-09-30T00:00:36.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091843b523a96be27-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.119 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.119 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.119 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.119 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.119 DEBUG:boto:Data: 2023-09-30T00:00:36.119 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.119 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.119 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.119 DEBUG:boto:Params: {} 2023-09-30T00:00:36.120 DEBUG:boto:Token: None 2023-09-30T00:00:36.120 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.120 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aad03c69ef1f77a-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.124 DEBUG:boto:path=/dmuhsu-72/obj145 2023-09-30T00:00:36.124 DEBUG:boto:auth_path=/dmuhsu-72/obj145 2023-09-30T00:00:36.124 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.124 DEBUG:boto:Path: /dmuhsu-72/obj145 2023-09-30T00:00:36.124 DEBUG:boto:Data: 2023-09-30T00:00:36.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.124 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.124 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.124 DEBUG:boto:Params: {} 2023-09-30T00:00:36.125 DEBUG:boto:Token: None 2023-09-30T00:00:36.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj145 2023-09-30T00:00:36.125 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zkcfriRZP8ID2z3dO7VKKjeKJ9c= 2023-09-30T00:00:36.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zkcfriRZP8ID2z3dO7VKKjeKJ9c='} 2023-09-30T00:00:36.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e38c401c8b60f3a-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.130 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:36.178 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:36.240 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:36.241 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:36.250 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 25 2023-09-30T00:00:36.340 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.340 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.340 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.340 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.340 DEBUG:boto:Data: 2023-09-30T00:00:36.340 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.340 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.340 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.341 DEBUG:boto:Params: {} 2023-09-30T00:00:36.341 DEBUG:boto:Token: None 2023-09-30T00:00:36.341 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.341 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d63d949b7426de-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.345 DEBUG:boto:path=/dmuhsu-72/obj31 2023-09-30T00:00:36.345 DEBUG:boto:auth_path=/dmuhsu-72/obj31 2023-09-30T00:00:36.345 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.345 DEBUG:boto:Path: /dmuhsu-72/obj31 2023-09-30T00:00:36.345 DEBUG:boto:Data: 2023-09-30T00:00:36.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.346 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.346 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.346 DEBUG:boto:Params: {} 2023-09-30T00:00:36.346 DEBUG:boto:Token: None 2023-09-30T00:00:36.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj31 2023-09-30T00:00:36.346 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3GIN1CM7rzkMvtrhnzvHxa+tvHk= 2023-09-30T00:00:36.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3GIN1CM7rzkMvtrhnzvHxa+tvHk='} 2023-09-30T00:00:36.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000062d360b6bb9c9cf-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.395 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.395 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.395 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.395 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.395 DEBUG:boto:Data: 2023-09-30T00:00:36.395 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.395 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.396 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.396 DEBUG:boto:Params: {} 2023-09-30T00:00:36.396 DEBUG:boto:Token: None 2023-09-30T00:00:36.396 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.396 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c64b3f40d321a607-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.400 DEBUG:boto:path=/dmuhsu-72/obj62 2023-09-30T00:00:36.400 DEBUG:boto:auth_path=/dmuhsu-72/obj62 2023-09-30T00:00:36.400 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.400 DEBUG:boto:Path: /dmuhsu-72/obj62 2023-09-30T00:00:36.400 DEBUG:boto:Data: 2023-09-30T00:00:36.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.400 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.400 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.400 DEBUG:boto:Params: {} 2023-09-30T00:00:36.401 DEBUG:boto:Token: None 2023-09-30T00:00:36.401 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj62 2023-09-30T00:00:36.401 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9/m3ArO3OpI83AME6zkpjF+6+Ls= 2023-09-30T00:00:36.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9/m3ArO3OpI83AME6zkpjF+6+Ls='} 2023-09-30T00:00:36.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000527105ad2a94862a-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.406 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.407 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.407 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.407 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.407 DEBUG:boto:Data: 2023-09-30T00:00:36.407 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.407 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.407 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.407 DEBUG:boto:Params: {} 2023-09-30T00:00:36.407 DEBUG:boto:Token: None 2023-09-30T00:00:36.407 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.408 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009578fbc01dce6a5c-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.411 DEBUG:boto:path=/dmuhsu-72/obj93 2023-09-30T00:00:36.411 DEBUG:boto:auth_path=/dmuhsu-72/obj93 2023-09-30T00:00:36.411 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.411 DEBUG:boto:Path: /dmuhsu-72/obj93 2023-09-30T00:00:36.411 DEBUG:boto:Data: 2023-09-30T00:00:36.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.411 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.411 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.411 DEBUG:boto:Params: {} 2023-09-30T00:00:36.412 DEBUG:boto:Token: None 2023-09-30T00:00:36.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj93 2023-09-30T00:00:36.412 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YB7VDPdWCQSdH/FZyTZJX9tc+XY= 2023-09-30T00:00:36.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YB7VDPdWCQSdH/FZyTZJX9tc+XY='} 2023-09-30T00:00:36.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a30f7f7210cb727-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.455 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.455 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.455 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.455 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.455 DEBUG:boto:Data: 2023-09-30T00:00:36.455 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.456 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.456 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.456 DEBUG:boto:Params: {} 2023-09-30T00:00:36.456 DEBUG:boto:Token: None 2023-09-30T00:00:36.456 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000992c76c587a3ce3f-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.459 DEBUG:boto:path=/dmuhsu-72/obj124 2023-09-30T00:00:36.460 DEBUG:boto:auth_path=/dmuhsu-72/obj124 2023-09-30T00:00:36.460 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.460 DEBUG:boto:Path: /dmuhsu-72/obj124 2023-09-30T00:00:36.460 DEBUG:boto:Data: 2023-09-30T00:00:36.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.460 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.460 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.460 DEBUG:boto:Params: {} 2023-09-30T00:00:36.460 DEBUG:boto:Token: None 2023-09-30T00:00:36.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj124 2023-09-30T00:00:36.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hWnRPRo5YGevb8qvYRD7ns70aXo= 2023-09-30T00:00:36.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hWnRPRo5YGevb8qvYRD7ns70aXo='} 2023-09-30T00:00:36.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f10b6af485267cf-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.466 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.467 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.467 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.467 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.467 DEBUG:boto:Data: 2023-09-30T00:00:36.467 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.467 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.467 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.467 DEBUG:boto:Params: {} 2023-09-30T00:00:36.467 DEBUG:boto:Token: None 2023-09-30T00:00:36.467 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.468 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c6a50d83e1a974a-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.471 DEBUG:boto:path=/dmuhsu-72/obj155 2023-09-30T00:00:36.471 DEBUG:boto:auth_path=/dmuhsu-72/obj155 2023-09-30T00:00:36.471 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.471 DEBUG:boto:Path: /dmuhsu-72/obj155 2023-09-30T00:00:36.471 DEBUG:boto:Data: 2023-09-30T00:00:36.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.472 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.472 DEBUG:boto:Params: {} 2023-09-30T00:00:36.472 DEBUG:boto:Token: None 2023-09-30T00:00:36.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj155 2023-09-30T00:00:36.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MrUFwIKF2WSMLH+XWN/TPG7TuXE= 2023-09-30T00:00:36.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MrUFwIKF2WSMLH+XWN/TPG7TuXE='} 2023-09-30T00:00:36.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dbd9466fbbff7b4-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.519 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:36.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:36.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:36.648 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:00:36.648 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-72 2023-09-30T00:00:36.659 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 30 2023-09-30T00:00:36.758 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.759 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.759 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.759 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.759 DEBUG:boto:Data: 2023-09-30T00:00:36.759 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.759 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.759 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.759 DEBUG:boto:Params: {} 2023-09-30T00:00:36.759 DEBUG:boto:Token: None 2023-09-30T00:00:36.760 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.760 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d31e6df899155c5f-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.764 DEBUG:boto:path=/dmuhsu-72/obj37 2023-09-30T00:00:36.764 DEBUG:boto:auth_path=/dmuhsu-72/obj37 2023-09-30T00:00:36.764 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.764 DEBUG:boto:Path: /dmuhsu-72/obj37 2023-09-30T00:00:36.764 DEBUG:boto:Data: 2023-09-30T00:00:36.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.764 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.764 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.765 DEBUG:boto:Params: {} 2023-09-30T00:00:36.765 DEBUG:boto:Token: None 2023-09-30T00:00:36.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj37 2023-09-30T00:00:36.765 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1n17JKPLoS+fnZpPm8On+gbtR24= 2023-09-30T00:00:36.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1n17JKPLoS+fnZpPm8On+gbtR24='} 2023-09-30T00:00:36.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f98ee9fe4510b12f-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.770 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.771 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.771 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.771 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.771 DEBUG:boto:Data: 2023-09-30T00:00:36.771 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.771 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.771 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.771 DEBUG:boto:Params: {} 2023-09-30T00:00:36.771 DEBUG:boto:Token: None 2023-09-30T00:00:36.772 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.772 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000346ad7c0d34ad9da-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.776 DEBUG:boto:path=/dmuhsu-72/obj74 2023-09-30T00:00:36.776 DEBUG:boto:auth_path=/dmuhsu-72/obj74 2023-09-30T00:00:36.776 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.776 DEBUG:boto:Path: /dmuhsu-72/obj74 2023-09-30T00:00:36.776 DEBUG:boto:Data: 2023-09-30T00:00:36.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.777 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.777 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.777 DEBUG:boto:Params: {} 2023-09-30T00:00:36.777 DEBUG:boto:Token: None 2023-09-30T00:00:36.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj74 2023-09-30T00:00:36.777 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+f9PZDn9QJhOnmR8rucqcDrRbUU= 2023-09-30T00:00:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+f9PZDn9QJhOnmR8rucqcDrRbUU='} 2023-09-30T00:00:36.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c648c1ae8c27a3d1-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.827 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.827 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.827 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.827 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.827 DEBUG:boto:Data: 2023-09-30T00:00:36.828 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.828 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.828 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.828 DEBUG:boto:Params: {} 2023-09-30T00:00:36.828 DEBUG:boto:Token: None 2023-09-30T00:00:36.828 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.828 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f6393470731d38-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.832 DEBUG:boto:path=/dmuhsu-72/obj111 2023-09-30T00:00:36.832 DEBUG:boto:auth_path=/dmuhsu-72/obj111 2023-09-30T00:00:36.832 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.832 DEBUG:boto:Path: /dmuhsu-72/obj111 2023-09-30T00:00:36.833 DEBUG:boto:Data: 2023-09-30T00:00:36.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.833 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.833 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.833 DEBUG:boto:Params: {} 2023-09-30T00:00:36.834 DEBUG:boto:Token: None 2023-09-30T00:00:36.834 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj111 2023-09-30T00:00:36.834 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oNSY0MudyCdgJA8M4eDkbkHDrss= 2023-09-30T00:00:36.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oNSY0MudyCdgJA8M4eDkbkHDrss='} 2023-09-30T00:00:36.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e949118bd33d85f-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.839 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.839 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.839 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.840 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.840 DEBUG:boto:Data: 2023-09-30T00:00:36.840 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.840 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.840 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.840 DEBUG:boto:Params: {} 2023-09-30T00:00:36.840 DEBUG:boto:Token: None 2023-09-30T00:00:36.840 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.840 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebbe027de036f65e-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.844 DEBUG:boto:path=/dmuhsu-72/obj148 2023-09-30T00:00:36.844 DEBUG:boto:auth_path=/dmuhsu-72/obj148 2023-09-30T00:00:36.845 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.845 DEBUG:boto:Path: /dmuhsu-72/obj148 2023-09-30T00:00:36.845 DEBUG:boto:Data: 2023-09-30T00:00:36.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.845 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.845 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.845 DEBUG:boto:Params: {} 2023-09-30T00:00:36.845 DEBUG:boto:Token: None 2023-09-30T00:00:36.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj148 2023-09-30T00:00:36.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5Xw2uSlZNdfX6V/JBrE4VsGzsxY= 2023-09-30T00:00:36.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5Xw2uSlZNdfX6V/JBrE4VsGzsxY='} 2023-09-30T00:00:36.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1c8787e799f5a35-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.891 DEBUG:boto:path=/dmuhsu-72/ 2023-09-30T00:00:36.891 DEBUG:boto:auth_path=/dmuhsu-72/ 2023-09-30T00:00:36.891 DEBUG:boto:Method: HEAD 2023-09-30T00:00:36.891 DEBUG:boto:Path: /dmuhsu-72/ 2023-09-30T00:00:36.891 DEBUG:boto:Data: 2023-09-30T00:00:36.891 DEBUG:boto:Headers: {} 2023-09-30T00:00:36.891 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.892 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.892 DEBUG:boto:Params: {} 2023-09-30T00:00:36.892 DEBUG:boto:Token: None 2023-09-30T00:00:36.892 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/ 2023-09-30T00:00:36.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:00:36.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:00:36.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f678851ceb02209-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.896 DEBUG:boto:path=/dmuhsu-72/obj185 2023-09-30T00:00:36.896 DEBUG:boto:auth_path=/dmuhsu-72/obj185 2023-09-30T00:00:36.896 DEBUG:boto:Method: PUT 2023-09-30T00:00:36.896 DEBUG:boto:Path: /dmuhsu-72/obj185 2023-09-30T00:00:36.896 DEBUG:boto:Data: 2023-09-30T00:00:36.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:00:36.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:00:36.897 DEBUG:boto:Port: 8000 2023-09-30T00:00:36.897 DEBUG:boto:Params: {} 2023-09-30T00:00:36.897 DEBUG:boto:Token: None 2023-09-30T00:00:36.897 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:00:36 GMT /dmuhsu-72/obj185 2023-09-30T00:00:36.897 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9tmi15GudFOSUpnk7RcDwoUTrHI= 2023-09-30T00:00:36.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9tmi15GudFOSUpnk7RcDwoUTrHI='} 2023-09-30T00:00:36.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097134474879bcdae-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:00:36.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:00:36.903 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-30T00:00:36.953 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:36.954 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:36.986 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:36.985+0000 7fc272815880 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-30T00:00:36.987 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:00:36.987+0000 7fc272815880 5 RGW-SYNC:data: acquired data sync status lease 2023-09-30T00:00:37.009 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-30T00:00:37.009 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-30T00:00:37.011 INFO:tasks.rgw.c2.client.0:Started 2023-09-30T00:00:37.011 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:00:37.028 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:00:37.028 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-30T00:00:37.029 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-30T00:00:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-30T00:00:37.030 INFO:teuthology.orchestra.run.smithi184.stderr:curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-30T00:00:38.030 DEBUG:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:00:38.053 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:00:38.053 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2023-09-30T00:00:38.055 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-30T00:00:38.055 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2023-09-30T00:00:38.056 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-72 2023-09-30T00:00:38.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:38.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:38.056 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:00:38.153 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:38.153 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:38.187 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-72 2023-09-30T00:00:38.202 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-72 2023-09-30T00:00:38.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:38.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:38.202 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:00:38.274 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:38.275 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:38.330 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:38.330+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:00:40.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-30T00:00:43.331 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:43.331+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:00:48.332 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:48.332+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:00:53.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:53.334+0000 7fa722c19880 1 bucket sync caught up with source: 2023-09-30T00:00:53.335 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , 00000000002.62.6, , , , , , 00000000002.85.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.42.6, , , , 00000000001.49.6] 2023-09-30T00:00:53.335 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , 00000000002.62.6, , , , , , 00000000002.85.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.42.6, , , , 00000000001.49.6] 2023-09-30T00:00:53.335 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:53.334+0000 7fa722c19880 0 bucket checkpoint complete 2023-09-30T00:00:53.353 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-72 2023-09-30T00:00:53.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:53.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:00:53.354 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:00:53.406 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:00:53.407 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:00:53.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:53.444+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:00:58.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:00:58.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:03.445 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:03.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:08.446 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:08.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:13.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:13.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:18.447 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:18.446+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:23.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:23.447+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:28.448 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:28.448+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:33.449 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:33.449+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:38.450 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:38.449+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:43.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:43.450+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:48.451 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:48.451+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:53.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:53.451+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:01:58.452 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:01:58.452+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:03.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:03.453+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:08.453 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:08.452+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:13.454 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:13.453+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:18.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:18.454+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:23.455 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:23.455+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:28.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:28.456+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:33.456 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:33.456+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:38.457 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:38.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:43.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:43.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:48.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:48.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:53.458 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:53.458+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:02:58.459 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:02:58.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:03.460 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:03.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:08.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:08.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:13.461 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:13.460+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:18.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:18.461+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:23.462 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:23.462+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:28.463 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:28.463+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:33.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:33.463+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:38.464 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:38.464+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:43.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:43.464+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:48.465 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:48.465+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:53.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:53.466+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:03:58.466 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:03:58.466+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:03.467 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:03.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:08.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:08.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:13.468 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:13.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:18.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:18.468+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:23.469 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:23.469+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:28.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:28.470+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:33.470 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:33.470+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:38.471 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:38.471+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:43.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:43.471+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:48.472 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:48.472+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:53.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:53.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:04:58.473 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:04:58.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:03.474 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:03.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:08.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:08.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:13.475 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:13.474+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:18.476 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:18.475+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:23.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:23.476+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:28.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:28.477+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:33.477 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:33.477+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:38.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:38.478+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:43.478 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:43.478+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:05:48.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:48.479+0000 7f05c5729880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:05:48.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:48.479+0000 7f05c5729880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:05:48.479 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:48.479+0000 7f05c5729880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:05:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-30T00:05:48.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2023-09-30T00:05:48.503 DEBUG:boto:path=/dmuhsu-73/ 2023-09-30T00:05:48.503 DEBUG:boto:auth_path=/dmuhsu-73/ 2023-09-30T00:05:48.503 DEBUG:boto:Method: PUT 2023-09-30T00:05:48.503 DEBUG:boto:Path: /dmuhsu-73/ 2023-09-30T00:05:48.504 DEBUG:boto:Data: 2023-09-30T00:05:48.504 DEBUG:boto:Headers: {} 2023-09-30T00:05:48.504 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:05:48.504 DEBUG:boto:Port: 8000 2023-09-30T00:05:48.504 DEBUG:boto:Params: {} 2023-09-30T00:05:48.504 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:05:48.504 DEBUG:boto:Token: None 2023-09-30T00:05:48.505 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:05:48 GMT /dmuhsu-73/ 2023-09-30T00:05:48.505 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jqTAnHiOLVIak3kbgGCP9jYORFI= 2023-09-30T00:05:48.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jqTAnHiOLVIak3kbgGCP9jYORFI=', 'Content-Length': '0'} 2023-09-30T00:05:48.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000feb0cf4c1933e8ff-006517665c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:05:48.535 DEBUG:rgw_multi.tests:created bucket=dmuhsu-73 2023-09-30T00:05:48.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.536 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:05:48.609 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:48.609 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:48.645 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:05:48.666 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:05:48.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.666 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:05:48.666 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:05:48.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:05:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:05:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:05:48.668 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:05:48.533216Z" 2023-09-30T00:05:48.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:05:48.669 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:05:48.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.669 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:05:48.670 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:05:48.670 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:05:48.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.671 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:05:48.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:48.725 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:48.754 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:05:48.754+0000 7f837c80f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:05:48.755 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:05:48.755+0000 7f837c80f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:05:48.769 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:05:48.769 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:05:48.769 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:05:48.769 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:05:48.770 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:05:48.771 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:05:48.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:05:48.773 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:05:48.774 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:05:48.775 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:05:48.776 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:05:48.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032348.533216_654.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:05:48.533216Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:05:48Z" } 2023-09-30T00:05:48.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032348.533216_654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:05:48.533216Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:05:48.777 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:05:48.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:05:48.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:05:48.777 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:05:48.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:05:48.778 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:05:48.824 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:48.824 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:48.863 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:48.863+0000 7fb1a2612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:05:48.864 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:48.864+0000 7fb1a2612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:05:48.878 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:05:48.878 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:05:48.878 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:05:48.878 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:05:48.879 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:05:48.881 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:05:48.882 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:05:48.883 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:05:48.884 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:05:48.885 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:05:48.885 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:05:48.885 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:05:48.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032348.533216_654.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:05:48.533216Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:05:48Z" } 2023-09-30T00:05:48.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032348.533216_654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:05:48.533216Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:05:48.886 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:05:48.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:05:48.886 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:05:48.886 DEBUG:boto:path=/dmuhsu-73/ 2023-09-30T00:05:48.887 DEBUG:boto:auth_path=/dmuhsu-73/ 2023-09-30T00:05:48.887 DEBUG:boto:Method: HEAD 2023-09-30T00:05:48.887 DEBUG:boto:Path: /dmuhsu-73/ 2023-09-30T00:05:48.887 DEBUG:boto:Data: 2023-09-30T00:05:48.887 DEBUG:boto:Headers: {} 2023-09-30T00:05:48.887 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:05:48.887 DEBUG:boto:Port: 8000 2023-09-30T00:05:48.887 DEBUG:boto:Params: {} 2023-09-30T00:05:48.887 DEBUG:boto:Token: None 2023-09-30T00:05:48.888 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:05:48 GMT /dmuhsu-73/ 2023-09-30T00:05:48.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Igh6JQfPZ/pQlpTfEGUDStbNbMM= 2023-09-30T00:05:48.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Igh6JQfPZ/pQlpTfEGUDStbNbMM=', 'Content-Length': '0'} 2023-09-30T00:05:48.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e22b7a8fa05f901-006517665c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:05:48.890 DEBUG:boto:path=/dmuhsu-73/dummy 2023-09-30T00:05:48.890 DEBUG:boto:auth_path=/dmuhsu-73/dummy 2023-09-30T00:05:48.890 DEBUG:boto:Method: PUT 2023-09-30T00:05:48.891 DEBUG:boto:Path: /dmuhsu-73/dummy 2023-09-30T00:05:48.891 DEBUG:boto:Data: 2023-09-30T00:05:48.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:05:48.891 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:05:48.891 DEBUG:boto:Port: 8000 2023-09-30T00:05:48.891 DEBUG:boto:Params: {} 2023-09-30T00:05:48.891 DEBUG:boto:Token: None 2023-09-30T00:05:48.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:05:48 GMT /dmuhsu-73/dummy 2023-09-30T00:05:48.891 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TRM6zsrbZzfMREXcssQ8RdrdRqM= 2023-09-30T00:05:48.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TRM6zsrbZzfMREXcssQ8RdrdRqM='} 2023-09-30T00:05:48.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000965184032a20593e-006517665c-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:05:48.898 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-73 2023-09-30T00:05:48.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:48.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:48.899 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:05:48.952 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:48.953 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:48.983 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-73 2023-09-30T00:05:48.998 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-73 2023-09-30T00:05:48.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:48.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:48.998 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:05:49.050 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:49.051 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:49.092 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:49.092+0000 7f74c67a2880 1 waiting to reach incremental sync.. 2023-09-30T00:05:54.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:54.093+0000 7f74c67a2880 1 bucket sync caught up with source: 2023-09-30T00:05:54.094 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000001.65.6, , , , , ] 2023-09-30T00:05:54.094 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000001.65.6, , , , , ] 2023-09-30T00:05:54.094 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:54.093+0000 7f74c67a2880 0 bucket checkpoint complete 2023-09-30T00:05:54.114 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-73 2023-09-30T00:05:54.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:54.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:05:54.115 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:05:54.170 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:05:54.171 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:05:54.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:54.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:05:59.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:05:59.224+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:04.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:04.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:09.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:09.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:14.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:14.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:19.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:19.225+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:24.225+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:29.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:29.226+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:34.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:34.226+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:39.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:39.227+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:44.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:44.227+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:49.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:49.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:54.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:54.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:06:59.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:06:59.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:04.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:04.229+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:09.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:09.229+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:14.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:14.231+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:19.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:19.232+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:24.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:24.232+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:29.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:29.233+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:34.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:34.233+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:39.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:39.234+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:44.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:44.234+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:49.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:49.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:54.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:54.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:07:59.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:07:59.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:04.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:04.236+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:09.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:09.236+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:14.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:14.238+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:19.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:19.238+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:24.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:24.239+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:29.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:29.239+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:34.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:34.240+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:39.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:39.240+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:44.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:44.241+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:49.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:49.241+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:54.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:54.242+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:08:59.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:08:59.242+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:04.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:04.243+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:09.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:09.243+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:14.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:14.245+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:19.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:19.245+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:24.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:24.246+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:29.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:29.247+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:34.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:34.247+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:39.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:39.248+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:44.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:44.248+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:49.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:49.249+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:54.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:54.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:09:59.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:09:59.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:04.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:04.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:09.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:09.251+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:14.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:14.252+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:19.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:19.252+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:24.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:24.253+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:29.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:29.254+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:34.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:34.254+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:39.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:39.255+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:44.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:44.256+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:10:49.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:49.256+0000 7f0500d29880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:10:49.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:49.256+0000 7f0500d29880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:10:49.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:49.256+0000 7f0500d29880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:10:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-30T00:10:49.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2023-09-30T00:10:49.280 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:49.280 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:49.281 DEBUG:boto:Method: PUT 2023-09-30T00:10:49.281 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:49.281 DEBUG:boto:Data: 2023-09-30T00:10:49.281 DEBUG:boto:Headers: {} 2023-09-30T00:10:49.281 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:49.281 DEBUG:boto:Port: 8000 2023-09-30T00:10:49.281 DEBUG:boto:Params: {} 2023-09-30T00:10:49.281 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:10:49.282 DEBUG:boto:Token: None 2023-09-30T00:10:49.282 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:10:49 GMT /dmuhsu-74/ 2023-09-30T00:10:49.282 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iGA8db0jrM4WVV+w5uzviKZkj9M= 2023-09-30T00:10:49.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iGA8db0jrM4WVV+w5uzviKZkj9M=', 'Content-Length': '0'} 2023-09-30T00:10:49.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cc8f0566d11ea716-0065176789-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:49.321 DEBUG:rgw_multi.tests:created bucket=dmuhsu-74 2023-09-30T00:10:49.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:49.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:49.322 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:10:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:49.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:49.417 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:10:49.431 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:49.432 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:10:49.319298Z" 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:49.433 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:10:49.434 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:49.434 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:10:49.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:49.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:49.435 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:10:49.487 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:49.487 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:49.522 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:10:49.522+0000 7f03fd210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:49.523 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:10:49.523+0000 7f03fd210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:49.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:10:49.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:49.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:10:49.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032649.310028_655.1", 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:49.541 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:10:49.310028Z", 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:49.542 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:10:49.543 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:10:49.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:49.544 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:10:49Z" 2023-09-30T00:10:49.544 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:10:49.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032649.310028_655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:10:49.310028Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:10:49Z" } 2023-09-30T00:10:49.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.310028_655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.310028Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:10:49.545 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:49.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.310028_655.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:49.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696032649.319298_658.1 target=1_1696032649.310028_655.1 2023-09-30T00:10:54.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:54.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:54.547 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:10:54.599 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:54.600 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:54.632 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:10:54.632+0000 7fc1f08c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:54.633 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:10:54.633+0000 7fc1f08c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:54.647 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.648 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:10:54.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:54.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:54.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:10:54.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:10:54.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:10:54.654 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:10:54.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:10:54.654 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:10:54.654 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:10:54.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032649.319298_658.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:10:49.319298Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:10:54Z" } 2023-09-30T00:10:54.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.319298_658.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.319298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:10:54.655 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:54.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:54.655 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:10:54.655 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:10:54.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:54.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:10:54.656 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:10:54.708 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:54.708 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:54.764 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:54.763+0000 7f1536329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:54.765 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:54.764+0000 7f1536329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:10:54.791 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:10:54.792 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:10:54.793 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:10:54.794 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:10:54.795 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:10:54.796 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:10:54.797 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:10:54.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032649.319298_658.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:10:49.319298Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:10:54Z" } 2023-09-30T00:10:54.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.319298_658.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.319298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:10:54.798 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:54.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:10:54.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:10:54.799 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:54.799 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:54.799 DEBUG:boto:Method: HEAD 2023-09-30T00:10:54.799 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:54.799 DEBUG:boto:Data: 2023-09-30T00:10:54.799 DEBUG:boto:Headers: {} 2023-09-30T00:10:54.799 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.800 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.800 DEBUG:boto:Params: {} 2023-09-30T00:10:54.800 DEBUG:boto:Token: None 2023-09-30T00:10:54.800 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/ 2023-09-30T00:10:54.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:10:54.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:10:54.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037e46e668c8719c1-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.803 DEBUG:boto:path=/dmuhsu-74/obj11 2023-09-30T00:10:54.803 DEBUG:boto:auth_path=/dmuhsu-74/obj11 2023-09-30T00:10:54.803 DEBUG:boto:Method: PUT 2023-09-30T00:10:54.803 DEBUG:boto:Path: /dmuhsu-74/obj11 2023-09-30T00:10:54.803 DEBUG:boto:Data: 2023-09-30T00:10:54.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:54.804 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.804 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.804 DEBUG:boto:Params: {} 2023-09-30T00:10:54.804 DEBUG:boto:Token: None 2023-09-30T00:10:54.804 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/obj11 2023-09-30T00:10:54.804 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:62qD4aFLwtORPmed+ihB2QWdTWA= 2023-09-30T00:10:54.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:62qD4aFLwtORPmed+ihB2QWdTWA='} 2023-09-30T00:10:54.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003be97d62cca8a85e-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.812 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:54.813 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:54.813 DEBUG:boto:Method: HEAD 2023-09-30T00:10:54.813 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:54.813 DEBUG:boto:Data: 2023-09-30T00:10:54.813 DEBUG:boto:Headers: {} 2023-09-30T00:10:54.813 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.813 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.813 DEBUG:boto:Params: {} 2023-09-30T00:10:54.813 DEBUG:boto:Token: None 2023-09-30T00:10:54.814 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/ 2023-09-30T00:10:54.814 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:10:54.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:10:54.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6dade10bae95a9b-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.816 DEBUG:boto:path=/dmuhsu-74/obj22 2023-09-30T00:10:54.816 DEBUG:boto:auth_path=/dmuhsu-74/obj22 2023-09-30T00:10:54.816 DEBUG:boto:Method: PUT 2023-09-30T00:10:54.816 DEBUG:boto:Path: /dmuhsu-74/obj22 2023-09-30T00:10:54.816 DEBUG:boto:Data: 2023-09-30T00:10:54.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:54.817 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.817 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.817 DEBUG:boto:Params: {} 2023-09-30T00:10:54.817 DEBUG:boto:Token: None 2023-09-30T00:10:54.817 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/obj22 2023-09-30T00:10:54.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cIEnzzXxifeKXcZA4bAw4gMymV8= 2023-09-30T00:10:54.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cIEnzzXxifeKXcZA4bAw4gMymV8='} 2023-09-30T00:10:54.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003655181d3ff1910-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.821 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:54.821 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:54.821 DEBUG:boto:Method: HEAD 2023-09-30T00:10:54.821 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:54.822 DEBUG:boto:Data: 2023-09-30T00:10:54.822 DEBUG:boto:Headers: {} 2023-09-30T00:10:54.822 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.822 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.822 DEBUG:boto:Params: {} 2023-09-30T00:10:54.822 DEBUG:boto:Token: None 2023-09-30T00:10:54.822 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/ 2023-09-30T00:10:54.822 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:10:54.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:10:54.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f7faca0703e69da-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.825 DEBUG:boto:path=/dmuhsu-74/obj33 2023-09-30T00:10:54.825 DEBUG:boto:auth_path=/dmuhsu-74/obj33 2023-09-30T00:10:54.825 DEBUG:boto:Method: PUT 2023-09-30T00:10:54.825 DEBUG:boto:Path: /dmuhsu-74/obj33 2023-09-30T00:10:54.825 DEBUG:boto:Data: 2023-09-30T00:10:54.825 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:54.826 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.826 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.826 DEBUG:boto:Params: {} 2023-09-30T00:10:54.826 DEBUG:boto:Token: None 2023-09-30T00:10:54.826 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/obj33 2023-09-30T00:10:54.826 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y3BnmBBKMwxLtPAnZBF1UCk+Mro= 2023-09-30T00:10:54.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y3BnmBBKMwxLtPAnZBF1UCk+Mro='} 2023-09-30T00:10:54.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000140cc8d256761a90-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.875 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:54.875 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:54.875 DEBUG:boto:Method: HEAD 2023-09-30T00:10:54.875 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:54.875 DEBUG:boto:Data: 2023-09-30T00:10:54.875 DEBUG:boto:Headers: {} 2023-09-30T00:10:54.875 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.875 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.876 DEBUG:boto:Params: {} 2023-09-30T00:10:54.876 DEBUG:boto:Token: None 2023-09-30T00:10:54.876 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/ 2023-09-30T00:10:54.876 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:10:54.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:10:54.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000698b51ebcd32e513-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.878 DEBUG:boto:path=/dmuhsu-74/obj44 2023-09-30T00:10:54.878 DEBUG:boto:auth_path=/dmuhsu-74/obj44 2023-09-30T00:10:54.878 DEBUG:boto:Method: PUT 2023-09-30T00:10:54.879 DEBUG:boto:Path: /dmuhsu-74/obj44 2023-09-30T00:10:54.879 DEBUG:boto:Data: 2023-09-30T00:10:54.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:54.879 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.879 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.879 DEBUG:boto:Params: {} 2023-09-30T00:10:54.879 DEBUG:boto:Token: None 2023-09-30T00:10:54.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/obj44 2023-09-30T00:10:54.880 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dNhop5LZYrXV56j4QRZLh16CMx8= 2023-09-30T00:10:54.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dNhop5LZYrXV56j4QRZLh16CMx8='} 2023-09-30T00:10:54.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023b67c9634816d49-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.886 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:54.886 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:54.886 DEBUG:boto:Method: HEAD 2023-09-30T00:10:54.886 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:54.887 DEBUG:boto:Data: 2023-09-30T00:10:54.887 DEBUG:boto:Headers: {} 2023-09-30T00:10:54.887 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.887 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.887 DEBUG:boto:Params: {} 2023-09-30T00:10:54.887 DEBUG:boto:Token: None 2023-09-30T00:10:54.887 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/ 2023-09-30T00:10:54.887 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:10:54.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:10:54.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046828f503114ac0b-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.890 DEBUG:boto:path=/dmuhsu-74/obj55 2023-09-30T00:10:54.890 DEBUG:boto:auth_path=/dmuhsu-74/obj55 2023-09-30T00:10:54.890 DEBUG:boto:Method: PUT 2023-09-30T00:10:54.890 DEBUG:boto:Path: /dmuhsu-74/obj55 2023-09-30T00:10:54.890 DEBUG:boto:Data: 2023-09-30T00:10:54.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:54.890 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:54.891 DEBUG:boto:Port: 8000 2023-09-30T00:10:54.891 DEBUG:boto:Params: {} 2023-09-30T00:10:54.891 DEBUG:boto:Token: None 2023-09-30T00:10:54.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:54 GMT /dmuhsu-74/obj55 2023-09-30T00:10:54.891 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XbDaf2hZkBhOT2yqE0dAHuOJvlY= 2023-09-30T00:10:54.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XbDaf2hZkBhOT2yqE0dAHuOJvlY='} 2023-09-30T00:10:54.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3ea81b5e22c1449-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:54.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:54.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:54.895 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:54.943 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:54.944 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:54.994 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:54.994 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:54.998 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 5 2023-09-30T00:10:55.061 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.061 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.061 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.061 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.062 DEBUG:boto:Data: 2023-09-30T00:10:55.062 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.062 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.062 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.062 DEBUG:boto:Params: {} 2023-09-30T00:10:55.063 DEBUG:boto:Token: None 2023-09-30T00:10:55.063 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.063 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a3659c0c3a4c5dc-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.066 DEBUG:boto:path=/dmuhsu-74/obj17 2023-09-30T00:10:55.066 DEBUG:boto:auth_path=/dmuhsu-74/obj17 2023-09-30T00:10:55.067 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.067 DEBUG:boto:Path: /dmuhsu-74/obj17 2023-09-30T00:10:55.067 DEBUG:boto:Data: 2023-09-30T00:10:55.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.067 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.067 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.067 DEBUG:boto:Params: {} 2023-09-30T00:10:55.067 DEBUG:boto:Token: None 2023-09-30T00:10:55.068 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj17 2023-09-30T00:10:55.068 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Mdu8AGvY12qsW2JL8l4He1AbloA= 2023-09-30T00:10:55.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mdu8AGvY12qsW2JL8l4He1AbloA='} 2023-09-30T00:10:55.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d5f77f0582df06-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.077 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.077 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.077 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.077 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.077 DEBUG:boto:Data: 2023-09-30T00:10:55.078 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.078 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.078 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.078 DEBUG:boto:Params: {} 2023-09-30T00:10:55.078 DEBUG:boto:Token: None 2023-09-30T00:10:55.078 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.078 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071f2ccaa2eebe443-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.081 DEBUG:boto:path=/dmuhsu-74/obj34 2023-09-30T00:10:55.081 DEBUG:boto:auth_path=/dmuhsu-74/obj34 2023-09-30T00:10:55.081 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.081 DEBUG:boto:Path: /dmuhsu-74/obj34 2023-09-30T00:10:55.081 DEBUG:boto:Data: 2023-09-30T00:10:55.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.082 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.082 DEBUG:boto:Params: {} 2023-09-30T00:10:55.082 DEBUG:boto:Token: None 2023-09-30T00:10:55.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj34 2023-09-30T00:10:55.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CmLex/Glrcl1b7Y4jJLIkdKAPog= 2023-09-30T00:10:55.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CmLex/Glrcl1b7Y4jJLIkdKAPog='} 2023-09-30T00:10:55.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aca50992db9dbbc5-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.127 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.127 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.127 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.127 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.127 DEBUG:boto:Data: 2023-09-30T00:10:55.127 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.127 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.128 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.128 DEBUG:boto:Params: {} 2023-09-30T00:10:55.128 DEBUG:boto:Token: None 2023-09-30T00:10:55.128 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.128 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007be09e56bff61175-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.131 DEBUG:boto:path=/dmuhsu-74/obj51 2023-09-30T00:10:55.131 DEBUG:boto:auth_path=/dmuhsu-74/obj51 2023-09-30T00:10:55.131 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.131 DEBUG:boto:Path: /dmuhsu-74/obj51 2023-09-30T00:10:55.131 DEBUG:boto:Data: 2023-09-30T00:10:55.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.132 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.132 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.132 DEBUG:boto:Params: {} 2023-09-30T00:10:55.132 DEBUG:boto:Token: None 2023-09-30T00:10:55.132 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj51 2023-09-30T00:10:55.132 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eNwoszaQPPWWyvSdMuycrFwrSjo= 2023-09-30T00:10:55.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eNwoszaQPPWWyvSdMuycrFwrSjo='} 2023-09-30T00:10:55.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b3ea9de510a015a-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.137 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.137 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.137 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.137 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.138 DEBUG:boto:Data: 2023-09-30T00:10:55.138 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.138 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.138 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.138 DEBUG:boto:Params: {} 2023-09-30T00:10:55.138 DEBUG:boto:Token: None 2023-09-30T00:10:55.138 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.138 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075dd1ea23899c39d-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.141 DEBUG:boto:path=/dmuhsu-74/obj68 2023-09-30T00:10:55.141 DEBUG:boto:auth_path=/dmuhsu-74/obj68 2023-09-30T00:10:55.141 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.141 DEBUG:boto:Path: /dmuhsu-74/obj68 2023-09-30T00:10:55.141 DEBUG:boto:Data: 2023-09-30T00:10:55.141 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.142 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.142 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.142 DEBUG:boto:Params: {} 2023-09-30T00:10:55.142 DEBUG:boto:Token: None 2023-09-30T00:10:55.142 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj68 2023-09-30T00:10:55.142 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:flM1GxX6q+2G1K+XvFXOG2Ony2U= 2023-09-30T00:10:55.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:flM1GxX6q+2G1K+XvFXOG2Ony2U='} 2023-09-30T00:10:55.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb525a674a14ec58-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.187 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.187 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.187 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.187 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.187 DEBUG:boto:Data: 2023-09-30T00:10:55.187 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.187 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.188 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.188 DEBUG:boto:Params: {} 2023-09-30T00:10:55.188 DEBUG:boto:Token: None 2023-09-30T00:10:55.188 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.188 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ad021644141784c-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.191 DEBUG:boto:path=/dmuhsu-74/obj85 2023-09-30T00:10:55.191 DEBUG:boto:auth_path=/dmuhsu-74/obj85 2023-09-30T00:10:55.191 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.191 DEBUG:boto:Path: /dmuhsu-74/obj85 2023-09-30T00:10:55.192 DEBUG:boto:Data: 2023-09-30T00:10:55.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.192 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.192 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.192 DEBUG:boto:Params: {} 2023-09-30T00:10:55.192 DEBUG:boto:Token: None 2023-09-30T00:10:55.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj85 2023-09-30T00:10:55.193 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cyOr8rsuPYwoNQzrB7QuZd8Q9i4= 2023-09-30T00:10:55.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cyOr8rsuPYwoNQzrB7QuZd8Q9i4='} 2023-09-30T00:10:55.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e1cfde0ca4d4fa4-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.202 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:55.250 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:55.251 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:55.311 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:55.312 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:55.317 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 10 2023-09-30T00:10:55.393 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.393 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.394 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.394 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.394 DEBUG:boto:Data: 2023-09-30T00:10:55.394 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.394 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.394 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.394 DEBUG:boto:Params: {} 2023-09-30T00:10:55.394 DEBUG:boto:Token: None 2023-09-30T00:10:55.395 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.395 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bb0bd26d1f29b1-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.398 DEBUG:boto:path=/dmuhsu-74/obj19 2023-09-30T00:10:55.398 DEBUG:boto:auth_path=/dmuhsu-74/obj19 2023-09-30T00:10:55.398 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.399 DEBUG:boto:Path: /dmuhsu-74/obj19 2023-09-30T00:10:55.399 DEBUG:boto:Data: 2023-09-30T00:10:55.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.399 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.399 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.399 DEBUG:boto:Params: {} 2023-09-30T00:10:55.399 DEBUG:boto:Token: None 2023-09-30T00:10:55.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj19 2023-09-30T00:10:55.399 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:F1uvkGotPjKlAPEpcqvnPrSlgKk= 2023-09-30T00:10:55.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F1uvkGotPjKlAPEpcqvnPrSlgKk='} 2023-09-30T00:10:55.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022de351bc9691382-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.407 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.408 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.408 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.408 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.408 DEBUG:boto:Data: 2023-09-30T00:10:55.408 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.408 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.408 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.408 DEBUG:boto:Params: {} 2023-09-30T00:10:55.408 DEBUG:boto:Token: None 2023-09-30T00:10:55.409 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.409 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030b952396aa5e665-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.412 DEBUG:boto:path=/dmuhsu-74/obj38 2023-09-30T00:10:55.412 DEBUG:boto:auth_path=/dmuhsu-74/obj38 2023-09-30T00:10:55.412 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.412 DEBUG:boto:Path: /dmuhsu-74/obj38 2023-09-30T00:10:55.412 DEBUG:boto:Data: 2023-09-30T00:10:55.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.412 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.412 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.413 DEBUG:boto:Params: {} 2023-09-30T00:10:55.413 DEBUG:boto:Token: None 2023-09-30T00:10:55.413 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj38 2023-09-30T00:10:55.413 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9jjAv494blBc1ca2ZBawjJcnaOE= 2023-09-30T00:10:55.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9jjAv494blBc1ca2ZBawjJcnaOE='} 2023-09-30T00:10:55.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958efba8e61e9ecd-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.455 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.455 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.455 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.455 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.455 DEBUG:boto:Data: 2023-09-30T00:10:55.455 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.455 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.455 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.456 DEBUG:boto:Params: {} 2023-09-30T00:10:55.456 DEBUG:boto:Token: None 2023-09-30T00:10:55.456 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.456 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053b5356becb911ad-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.459 DEBUG:boto:path=/dmuhsu-74/obj57 2023-09-30T00:10:55.459 DEBUG:boto:auth_path=/dmuhsu-74/obj57 2023-09-30T00:10:55.459 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.459 DEBUG:boto:Path: /dmuhsu-74/obj57 2023-09-30T00:10:55.459 DEBUG:boto:Data: 2023-09-30T00:10:55.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.459 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.459 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.459 DEBUG:boto:Params: {} 2023-09-30T00:10:55.460 DEBUG:boto:Token: None 2023-09-30T00:10:55.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj57 2023-09-30T00:10:55.460 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DOkja9SSR3c2SavQ0FgNP52y8yk= 2023-09-30T00:10:55.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOkja9SSR3c2SavQ0FgNP52y8yk='} 2023-09-30T00:10:55.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6b6f2206ef42d9-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.465 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.465 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.465 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.466 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.466 DEBUG:boto:Data: 2023-09-30T00:10:55.466 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.466 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.466 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.466 DEBUG:boto:Params: {} 2023-09-30T00:10:55.466 DEBUG:boto:Token: None 2023-09-30T00:10:55.466 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.466 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6d70504e1bbcb2e-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.469 DEBUG:boto:path=/dmuhsu-74/obj76 2023-09-30T00:10:55.469 DEBUG:boto:auth_path=/dmuhsu-74/obj76 2023-09-30T00:10:55.469 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.469 DEBUG:boto:Path: /dmuhsu-74/obj76 2023-09-30T00:10:55.469 DEBUG:boto:Data: 2023-09-30T00:10:55.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.470 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.470 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.470 DEBUG:boto:Params: {} 2023-09-30T00:10:55.470 DEBUG:boto:Token: None 2023-09-30T00:10:55.470 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj76 2023-09-30T00:10:55.470 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E/qCIRFlWAwqUZcbrkjyp3X2UAE= 2023-09-30T00:10:55.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E/qCIRFlWAwqUZcbrkjyp3X2UAE='} 2023-09-30T00:10:55.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1c477f54c2fbef0-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.476 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.476 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.476 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.476 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.476 DEBUG:boto:Data: 2023-09-30T00:10:55.476 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.476 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.477 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.477 DEBUG:boto:Params: {} 2023-09-30T00:10:55.477 DEBUG:boto:Token: None 2023-09-30T00:10:55.477 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.477 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dff7f81ea1bc897-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.480 DEBUG:boto:path=/dmuhsu-74/obj95 2023-09-30T00:10:55.480 DEBUG:boto:auth_path=/dmuhsu-74/obj95 2023-09-30T00:10:55.480 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.480 DEBUG:boto:Path: /dmuhsu-74/obj95 2023-09-30T00:10:55.480 DEBUG:boto:Data: 2023-09-30T00:10:55.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.480 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.480 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.481 DEBUG:boto:Params: {} 2023-09-30T00:10:55.481 DEBUG:boto:Token: None 2023-09-30T00:10:55.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj95 2023-09-30T00:10:55.481 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:epFhBGwOK2lCOJigW0ReY43XnLI= 2023-09-30T00:10:55.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:epFhBGwOK2lCOJigW0ReY43XnLI='} 2023-09-30T00:10:55.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005026b041f76211e3-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.527 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:55.576 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:55.576 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:55.629 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:55.629 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:55.635 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 15 2023-09-30T00:10:55.712 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.712 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.712 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.712 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.712 DEBUG:boto:Data: 2023-09-30T00:10:55.712 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.712 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.713 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.713 DEBUG:boto:Params: {} 2023-09-30T00:10:55.713 DEBUG:boto:Token: None 2023-09-30T00:10:55.713 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.713 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5730d571318cb94-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.716 DEBUG:boto:path=/dmuhsu-74/obj23 2023-09-30T00:10:55.717 DEBUG:boto:auth_path=/dmuhsu-74/obj23 2023-09-30T00:10:55.717 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.717 DEBUG:boto:Path: /dmuhsu-74/obj23 2023-09-30T00:10:55.717 DEBUG:boto:Data: 2023-09-30T00:10:55.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.717 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.717 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.717 DEBUG:boto:Params: {} 2023-09-30T00:10:55.718 DEBUG:boto:Token: None 2023-09-30T00:10:55.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj23 2023-09-30T00:10:55.718 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r8bFKskplvTyfWZLIHxwzD9bniA= 2023-09-30T00:10:55.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r8bFKskplvTyfWZLIHxwzD9bniA='} 2023-09-30T00:10:55.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6f22926ad11510f-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.726 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.727 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.727 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.727 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.727 DEBUG:boto:Data: 2023-09-30T00:10:55.727 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.727 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.727 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.727 DEBUG:boto:Params: {} 2023-09-30T00:10:55.728 DEBUG:boto:Token: None 2023-09-30T00:10:55.728 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584af0e15eee66d8-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.731 DEBUG:boto:path=/dmuhsu-74/obj46 2023-09-30T00:10:55.731 DEBUG:boto:auth_path=/dmuhsu-74/obj46 2023-09-30T00:10:55.731 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.731 DEBUG:boto:Path: /dmuhsu-74/obj46 2023-09-30T00:10:55.731 DEBUG:boto:Data: 2023-09-30T00:10:55.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.732 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.732 DEBUG:boto:Params: {} 2023-09-30T00:10:55.732 DEBUG:boto:Token: None 2023-09-30T00:10:55.732 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj46 2023-09-30T00:10:55.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EenB18XUbXrQ+j0qqzaDjTdh3JY= 2023-09-30T00:10:55.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EenB18XUbXrQ+j0qqzaDjTdh3JY='} 2023-09-30T00:10:55.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088ba695be9cee062-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.775 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.775 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.776 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.776 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.776 DEBUG:boto:Data: 2023-09-30T00:10:55.776 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.776 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.777 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.777 DEBUG:boto:Params: {} 2023-09-30T00:10:55.777 DEBUG:boto:Token: None 2023-09-30T00:10:55.777 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.778 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2408a70ed83503a-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.782 DEBUG:boto:path=/dmuhsu-74/obj69 2023-09-30T00:10:55.782 DEBUG:boto:auth_path=/dmuhsu-74/obj69 2023-09-30T00:10:55.782 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.782 DEBUG:boto:Path: /dmuhsu-74/obj69 2023-09-30T00:10:55.783 DEBUG:boto:Data: 2023-09-30T00:10:55.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.783 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.783 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.784 DEBUG:boto:Params: {} 2023-09-30T00:10:55.784 DEBUG:boto:Token: None 2023-09-30T00:10:55.784 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj69 2023-09-30T00:10:55.784 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u/900EvxScsSO8u5/sG3u6wQ8mk= 2023-09-30T00:10:55.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u/900EvxScsSO8u5/sG3u6wQ8mk='} 2023-09-30T00:10:55.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033b5ef9f0c36594c-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.793 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.793 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.793 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.793 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.794 DEBUG:boto:Data: 2023-09-30T00:10:55.794 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.794 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.794 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.795 DEBUG:boto:Params: {} 2023-09-30T00:10:55.795 DEBUG:boto:Token: None 2023-09-30T00:10:55.795 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.795 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000402f2019ab0d9c4d-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.799 DEBUG:boto:path=/dmuhsu-74/obj92 2023-09-30T00:10:55.799 DEBUG:boto:auth_path=/dmuhsu-74/obj92 2023-09-30T00:10:55.799 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.800 DEBUG:boto:Path: /dmuhsu-74/obj92 2023-09-30T00:10:55.800 DEBUG:boto:Data: 2023-09-30T00:10:55.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.800 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.800 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.801 DEBUG:boto:Params: {} 2023-09-30T00:10:55.801 DEBUG:boto:Token: None 2023-09-30T00:10:55.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj92 2023-09-30T00:10:55.802 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X7h+9wl6rpQ1KuU8l68lx0nKgvA= 2023-09-30T00:10:55.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X7h+9wl6rpQ1KuU8l68lx0nKgvA='} 2023-09-30T00:10:55.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004f600865160ff78-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.847 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:55.847 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:55.847 DEBUG:boto:Method: HEAD 2023-09-30T00:10:55.848 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:55.848 DEBUG:boto:Data: 2023-09-30T00:10:55.848 DEBUG:boto:Headers: {} 2023-09-30T00:10:55.848 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.848 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.848 DEBUG:boto:Params: {} 2023-09-30T00:10:55.848 DEBUG:boto:Token: None 2023-09-30T00:10:55.848 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/ 2023-09-30T00:10:55.849 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:10:55.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:10:55.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fe0ca247e6be0de-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.852 DEBUG:boto:path=/dmuhsu-74/obj115 2023-09-30T00:10:55.852 DEBUG:boto:auth_path=/dmuhsu-74/obj115 2023-09-30T00:10:55.852 DEBUG:boto:Method: PUT 2023-09-30T00:10:55.852 DEBUG:boto:Path: /dmuhsu-74/obj115 2023-09-30T00:10:55.852 DEBUG:boto:Data: 2023-09-30T00:10:55.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:55.852 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:55.852 DEBUG:boto:Port: 8000 2023-09-30T00:10:55.853 DEBUG:boto:Params: {} 2023-09-30T00:10:55.853 DEBUG:boto:Token: None 2023-09-30T00:10:55.853 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:55 GMT /dmuhsu-74/obj115 2023-09-30T00:10:55.853 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GR6M4QtDGmKPnE7ibU655NoeuA0= 2023-09-30T00:10:55.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GR6M4QtDGmKPnE7ibU655NoeuA0='} 2023-09-30T00:10:55.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a719c8bc7f447d6-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:55.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:55.865 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:55.913 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:55.914 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:55.977 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:55.978 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:55.985 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 20 2023-09-30T00:10:56.076 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.076 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.076 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.076 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.077 DEBUG:boto:Data: 2023-09-30T00:10:56.077 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.077 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.077 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.077 DEBUG:boto:Params: {} 2023-09-30T00:10:56.077 DEBUG:boto:Token: None 2023-09-30T00:10:56.077 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.078 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7a4f54d7f385a76-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.081 DEBUG:boto:path=/dmuhsu-74/obj29 2023-09-30T00:10:56.081 DEBUG:boto:auth_path=/dmuhsu-74/obj29 2023-09-30T00:10:56.081 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.081 DEBUG:boto:Path: /dmuhsu-74/obj29 2023-09-30T00:10:56.081 DEBUG:boto:Data: 2023-09-30T00:10:56.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.082 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.082 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.082 DEBUG:boto:Params: {} 2023-09-30T00:10:56.082 DEBUG:boto:Token: None 2023-09-30T00:10:56.082 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj29 2023-09-30T00:10:56.082 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JWwEoavustB0FYr3HSY8evBIgPM= 2023-09-30T00:10:56.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JWwEoavustB0FYr3HSY8evBIgPM='} 2023-09-30T00:10:56.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079f15e79431f40f2-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.089 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.089 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.089 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.089 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.089 DEBUG:boto:Data: 2023-09-30T00:10:56.089 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.089 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.089 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.089 DEBUG:boto:Params: {} 2023-09-30T00:10:56.090 DEBUG:boto:Token: None 2023-09-30T00:10:56.090 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000427ce68922140965-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.093 DEBUG:boto:path=/dmuhsu-74/obj58 2023-09-30T00:10:56.093 DEBUG:boto:auth_path=/dmuhsu-74/obj58 2023-09-30T00:10:56.093 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.094 DEBUG:boto:Path: /dmuhsu-74/obj58 2023-09-30T00:10:56.094 DEBUG:boto:Data: 2023-09-30T00:10:56.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.094 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.094 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.094 DEBUG:boto:Params: {} 2023-09-30T00:10:56.094 DEBUG:boto:Token: None 2023-09-30T00:10:56.094 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj58 2023-09-30T00:10:56.095 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ugbH4/0NEDodc4sK/G6G+vGfqI4= 2023-09-30T00:10:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ugbH4/0NEDodc4sK/G6G+vGfqI4='} 2023-09-30T00:10:56.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e0ef541f7d45d9e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.139 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.139 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.139 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.139 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.139 DEBUG:boto:Data: 2023-09-30T00:10:56.139 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.139 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.140 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.140 DEBUG:boto:Params: {} 2023-09-30T00:10:56.140 DEBUG:boto:Token: None 2023-09-30T00:10:56.140 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.140 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de46d5d3ffe8bfc4-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.143 DEBUG:boto:path=/dmuhsu-74/obj87 2023-09-30T00:10:56.143 DEBUG:boto:auth_path=/dmuhsu-74/obj87 2023-09-30T00:10:56.143 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.144 DEBUG:boto:Path: /dmuhsu-74/obj87 2023-09-30T00:10:56.144 DEBUG:boto:Data: 2023-09-30T00:10:56.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.144 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.144 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.144 DEBUG:boto:Params: {} 2023-09-30T00:10:56.144 DEBUG:boto:Token: None 2023-09-30T00:10:56.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj87 2023-09-30T00:10:56.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jWG0LE8oAjYjNbD4BrYtlGLn+F4= 2023-09-30T00:10:56.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jWG0LE8oAjYjNbD4BrYtlGLn+F4='} 2023-09-30T00:10:56.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008683ed196478a9be-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.149 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.149 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.150 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.150 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.150 DEBUG:boto:Data: 2023-09-30T00:10:56.150 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.150 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.150 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.150 DEBUG:boto:Params: {} 2023-09-30T00:10:56.150 DEBUG:boto:Token: None 2023-09-30T00:10:56.150 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000babd2e9ac98b968a-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.154 DEBUG:boto:path=/dmuhsu-74/obj116 2023-09-30T00:10:56.154 DEBUG:boto:auth_path=/dmuhsu-74/obj116 2023-09-30T00:10:56.154 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.154 DEBUG:boto:Path: /dmuhsu-74/obj116 2023-09-30T00:10:56.154 DEBUG:boto:Data: 2023-09-30T00:10:56.154 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.154 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.155 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.155 DEBUG:boto:Params: {} 2023-09-30T00:10:56.155 DEBUG:boto:Token: None 2023-09-30T00:10:56.155 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj116 2023-09-30T00:10:56.155 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4opB2jlCZOlBP/JRJVTcczssziw= 2023-09-30T00:10:56.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4opB2jlCZOlBP/JRJVTcczssziw='} 2023-09-30T00:10:56.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096565618a305d254-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.203 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.203 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.203 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.203 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.203 DEBUG:boto:Data: 2023-09-30T00:10:56.203 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.204 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.204 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.204 DEBUG:boto:Params: {} 2023-09-30T00:10:56.204 DEBUG:boto:Token: None 2023-09-30T00:10:56.204 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c8c4363e740601-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.207 DEBUG:boto:path=/dmuhsu-74/obj145 2023-09-30T00:10:56.207 DEBUG:boto:auth_path=/dmuhsu-74/obj145 2023-09-30T00:10:56.208 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.208 DEBUG:boto:Path: /dmuhsu-74/obj145 2023-09-30T00:10:56.208 DEBUG:boto:Data: 2023-09-30T00:10:56.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.208 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.208 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.209 DEBUG:boto:Params: {} 2023-09-30T00:10:56.209 DEBUG:boto:Token: None 2023-09-30T00:10:56.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj145 2023-09-30T00:10:56.209 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kjEGs4Dirv1OchCWpXAmENEiadI= 2023-09-30T00:10:56.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kjEGs4Dirv1OchCWpXAmENEiadI='} 2023-09-30T00:10:56.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcfc04edd762a9f5-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:56.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:56.217 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:56.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:56.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:56.338 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:56.339 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:56.348 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 25 2023-09-30T00:10:56.446 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.446 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.447 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.447 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.447 DEBUG:boto:Data: 2023-09-30T00:10:56.447 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.447 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.447 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.447 DEBUG:boto:Params: {} 2023-09-30T00:10:56.447 DEBUG:boto:Token: None 2023-09-30T00:10:56.448 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.448 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc70947b34893ebe-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.451 DEBUG:boto:path=/dmuhsu-74/obj31 2023-09-30T00:10:56.451 DEBUG:boto:auth_path=/dmuhsu-74/obj31 2023-09-30T00:10:56.452 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.452 DEBUG:boto:Path: /dmuhsu-74/obj31 2023-09-30T00:10:56.452 DEBUG:boto:Data: 2023-09-30T00:10:56.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.452 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.452 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.452 DEBUG:boto:Params: {} 2023-09-30T00:10:56.452 DEBUG:boto:Token: None 2023-09-30T00:10:56.452 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj31 2023-09-30T00:10:56.453 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zPzWkmJlp3rHJG9wWiCWcHdDhQg= 2023-09-30T00:10:56.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zPzWkmJlp3rHJG9wWiCWcHdDhQg='} 2023-09-30T00:10:56.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958b14d965e1b84e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.495 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.495 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.495 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.495 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.495 DEBUG:boto:Data: 2023-09-30T00:10:56.495 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.496 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.496 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.496 DEBUG:boto:Params: {} 2023-09-30T00:10:56.496 DEBUG:boto:Token: None 2023-09-30T00:10:56.496 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.496 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084f259ac71f38546-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.500 DEBUG:boto:path=/dmuhsu-74/obj62 2023-09-30T00:10:56.500 DEBUG:boto:auth_path=/dmuhsu-74/obj62 2023-09-30T00:10:56.500 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.500 DEBUG:boto:Path: /dmuhsu-74/obj62 2023-09-30T00:10:56.500 DEBUG:boto:Data: 2023-09-30T00:10:56.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.500 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.500 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.500 DEBUG:boto:Params: {} 2023-09-30T00:10:56.501 DEBUG:boto:Token: None 2023-09-30T00:10:56.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj62 2023-09-30T00:10:56.501 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VJjOuOpePsYtoMIHm7aqTlQctrg= 2023-09-30T00:10:56.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VJjOuOpePsYtoMIHm7aqTlQctrg='} 2023-09-30T00:10:56.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3c105077b61b45c-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.512 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.512 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.512 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.512 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.512 DEBUG:boto:Data: 2023-09-30T00:10:56.513 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.513 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.513 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.513 DEBUG:boto:Params: {} 2023-09-30T00:10:56.513 DEBUG:boto:Token: None 2023-09-30T00:10:56.513 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbe3a2f46c3399cd-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.516 DEBUG:boto:path=/dmuhsu-74/obj93 2023-09-30T00:10:56.517 DEBUG:boto:auth_path=/dmuhsu-74/obj93 2023-09-30T00:10:56.517 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.517 DEBUG:boto:Path: /dmuhsu-74/obj93 2023-09-30T00:10:56.517 DEBUG:boto:Data: 2023-09-30T00:10:56.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.517 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.517 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.517 DEBUG:boto:Params: {} 2023-09-30T00:10:56.517 DEBUG:boto:Token: None 2023-09-30T00:10:56.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj93 2023-09-30T00:10:56.518 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Cc1K4ZIe1xgRux5FXs3ovky6HgU= 2023-09-30T00:10:56.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cc1K4ZIe1xgRux5FXs3ovky6HgU='} 2023-09-30T00:10:56.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014634d24b52e9242-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.563 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.563 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.563 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.563 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.563 DEBUG:boto:Data: 2023-09-30T00:10:56.564 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.564 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.564 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.564 DEBUG:boto:Params: {} 2023-09-30T00:10:56.564 DEBUG:boto:Token: None 2023-09-30T00:10:56.564 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.564 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000716183e72cb11250-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.568 DEBUG:boto:path=/dmuhsu-74/obj124 2023-09-30T00:10:56.568 DEBUG:boto:auth_path=/dmuhsu-74/obj124 2023-09-30T00:10:56.568 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.568 DEBUG:boto:Path: /dmuhsu-74/obj124 2023-09-30T00:10:56.568 DEBUG:boto:Data: 2023-09-30T00:10:56.568 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.568 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.568 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.568 DEBUG:boto:Params: {} 2023-09-30T00:10:56.569 DEBUG:boto:Token: None 2023-09-30T00:10:56.569 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj124 2023-09-30T00:10:56.569 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zaFaIdS845tn5V3w3I9HT959SS8= 2023-09-30T00:10:56.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zaFaIdS845tn5V3w3I9HT959SS8='} 2023-09-30T00:10:56.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009931a890b1a1212-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.577 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.577 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.577 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.577 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.577 DEBUG:boto:Data: 2023-09-30T00:10:56.577 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.578 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.578 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.578 DEBUG:boto:Params: {} 2023-09-30T00:10:56.578 DEBUG:boto:Token: None 2023-09-30T00:10:56.578 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.578 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be4a6bad169fb908-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.582 DEBUG:boto:path=/dmuhsu-74/obj155 2023-09-30T00:10:56.582 DEBUG:boto:auth_path=/dmuhsu-74/obj155 2023-09-30T00:10:56.582 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.582 DEBUG:boto:Path: /dmuhsu-74/obj155 2023-09-30T00:10:56.582 DEBUG:boto:Data: 2023-09-30T00:10:56.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.582 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.582 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.583 DEBUG:boto:Params: {} 2023-09-30T00:10:56.583 DEBUG:boto:Token: None 2023-09-30T00:10:56.583 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj155 2023-09-30T00:10:56.583 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZjhH4a+P0T14immorJyqo7/k2xM= 2023-09-30T00:10:56.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjhH4a+P0T14immorJyqo7/k2xM='} 2023-09-30T00:10:56.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a637e44425f97f57-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:56.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:10:56.627 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:10:56.676 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:56.676 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:56.735 INFO:teuthology.orchestra.run.smithi175.stdout:tenant: 2023-09-30T00:10:56.735 INFO:teuthology.orchestra.run.smithi175.stdout:bucket name: dmuhsu-74 2023-09-30T00:10:56.745 INFO:teuthology.orchestra.run.smithi175.stdout:total entries: 30 2023-09-30T00:10:56.872 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.872 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.872 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.872 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.872 DEBUG:boto:Data: 2023-09-30T00:10:56.872 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.872 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.873 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.873 DEBUG:boto:Params: {} 2023-09-30T00:10:56.873 DEBUG:boto:Token: None 2023-09-30T00:10:56.873 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.873 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a57aecc1bcb22b5c-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.877 DEBUG:boto:path=/dmuhsu-74/obj37 2023-09-30T00:10:56.878 DEBUG:boto:auth_path=/dmuhsu-74/obj37 2023-09-30T00:10:56.878 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.878 DEBUG:boto:Path: /dmuhsu-74/obj37 2023-09-30T00:10:56.878 DEBUG:boto:Data: 2023-09-30T00:10:56.878 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.878 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.878 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.878 DEBUG:boto:Params: {} 2023-09-30T00:10:56.879 DEBUG:boto:Token: None 2023-09-30T00:10:56.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj37 2023-09-30T00:10:56.879 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bByR5LTIxOaSVz2xFRbTjjNIMlk= 2023-09-30T00:10:56.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bByR5LTIxOaSVz2xFRbTjjNIMlk='} 2023-09-30T00:10:56.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c56bd7b2c0900c16-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.889 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.890 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.890 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.890 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.890 DEBUG:boto:Data: 2023-09-30T00:10:56.890 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.890 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.890 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.890 DEBUG:boto:Params: {} 2023-09-30T00:10:56.891 DEBUG:boto:Token: None 2023-09-30T00:10:56.891 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.891 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea297e0feb8462f3-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.895 DEBUG:boto:path=/dmuhsu-74/obj74 2023-09-30T00:10:56.895 DEBUG:boto:auth_path=/dmuhsu-74/obj74 2023-09-30T00:10:56.895 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.895 DEBUG:boto:Path: /dmuhsu-74/obj74 2023-09-30T00:10:56.896 DEBUG:boto:Data: 2023-09-30T00:10:56.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.896 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.896 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.896 DEBUG:boto:Params: {} 2023-09-30T00:10:56.896 DEBUG:boto:Token: None 2023-09-30T00:10:56.896 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj74 2023-09-30T00:10:56.896 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:miMmxuzbrCwo4PfAluxsJdf9jFY= 2023-09-30T00:10:56.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:miMmxuzbrCwo4PfAluxsJdf9jFY='} 2023-09-30T00:10:56.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7d85144b65cc9a-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.939 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.939 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.939 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.939 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.939 DEBUG:boto:Data: 2023-09-30T00:10:56.939 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.939 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.940 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.940 DEBUG:boto:Params: {} 2023-09-30T00:10:56.940 DEBUG:boto:Token: None 2023-09-30T00:10:56.940 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.940 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000467bb035aaed9edb-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.944 DEBUG:boto:path=/dmuhsu-74/obj111 2023-09-30T00:10:56.944 DEBUG:boto:auth_path=/dmuhsu-74/obj111 2023-09-30T00:10:56.944 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.944 DEBUG:boto:Path: /dmuhsu-74/obj111 2023-09-30T00:10:56.944 DEBUG:boto:Data: 2023-09-30T00:10:56.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.945 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.945 DEBUG:boto:Params: {} 2023-09-30T00:10:56.945 DEBUG:boto:Token: None 2023-09-30T00:10:56.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj111 2023-09-30T00:10:56.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6JsWMfCgf7hzfxDuDdBScoM1jcQ= 2023-09-30T00:10:56.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6JsWMfCgf7hzfxDuDdBScoM1jcQ='} 2023-09-30T00:10:56.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014402872af3ee0aa-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.950 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.950 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.951 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.951 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.951 DEBUG:boto:Data: 2023-09-30T00:10:56.951 DEBUG:boto:Headers: {} 2023-09-30T00:10:56.951 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.951 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.951 DEBUG:boto:Params: {} 2023-09-30T00:10:56.951 DEBUG:boto:Token: None 2023-09-30T00:10:56.952 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/ 2023-09-30T00:10:56.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:10:56.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:10:56.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ea6433bb077e6e7-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.955 DEBUG:boto:path=/dmuhsu-74/obj148 2023-09-30T00:10:56.956 DEBUG:boto:auth_path=/dmuhsu-74/obj148 2023-09-30T00:10:56.956 DEBUG:boto:Method: PUT 2023-09-30T00:10:56.956 DEBUG:boto:Path: /dmuhsu-74/obj148 2023-09-30T00:10:56.956 DEBUG:boto:Data: 2023-09-30T00:10:56.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:56.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:56.956 DEBUG:boto:Port: 8000 2023-09-30T00:10:56.956 DEBUG:boto:Params: {} 2023-09-30T00:10:56.957 DEBUG:boto:Token: None 2023-09-30T00:10:56.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:56 GMT /dmuhsu-74/obj148 2023-09-30T00:10:56.957 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1cs7JbQ498vi7xlZU0OPIZPbWTg= 2023-09-30T00:10:56.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1cs7JbQ498vi7xlZU0OPIZPbWTg='} 2023-09-30T00:10:56.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5f150af141037e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:56.999 DEBUG:boto:path=/dmuhsu-74/ 2023-09-30T00:10:56.999 DEBUG:boto:auth_path=/dmuhsu-74/ 2023-09-30T00:10:56.999 DEBUG:boto:Method: HEAD 2023-09-30T00:10:56.999 DEBUG:boto:Path: /dmuhsu-74/ 2023-09-30T00:10:56.999 DEBUG:boto:Data: 2023-09-30T00:10:57.000 DEBUG:boto:Headers: {} 2023-09-30T00:10:57.000 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:57.000 DEBUG:boto:Port: 8000 2023-09-30T00:10:57.000 DEBUG:boto:Params: {} 2023-09-30T00:10:57.000 DEBUG:boto:Token: None 2023-09-30T00:10:57.000 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:10:57 GMT /dmuhsu-74/ 2023-09-30T00:10:57.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wSOLnOLYQwxAyuE4MX42FmWyCPA= 2023-09-30T00:10:57.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wSOLnOLYQwxAyuE4MX42FmWyCPA=', 'Content-Length': '0'} 2023-09-30T00:10:57.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a353cd9a987232e1-0065176791-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:57.004 DEBUG:boto:path=/dmuhsu-74/obj185 2023-09-30T00:10:57.004 DEBUG:boto:auth_path=/dmuhsu-74/obj185 2023-09-30T00:10:57.004 DEBUG:boto:Method: PUT 2023-09-30T00:10:57.005 DEBUG:boto:Path: /dmuhsu-74/obj185 2023-09-30T00:10:57.005 DEBUG:boto:Data: 2023-09-30T00:10:57.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:10:57.005 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:10:57.005 DEBUG:boto:Port: 8000 2023-09-30T00:10:57.005 DEBUG:boto:Params: {} 2023-09-30T00:10:57.005 DEBUG:boto:Token: None 2023-09-30T00:10:57.006 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:10:57 GMT /dmuhsu-74/obj185 2023-09-30T00:10:57.006 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TnZFYxzcvGM+6G+lxULBW+FEhRA= 2023-09-30T00:10:57.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TnZFYxzcvGM+6G+lxULBW+FEhRA='} 2023-09-30T00:10:57.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000150c191bc658ed55-0065176791-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:10:57.011 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-74 2023-09-30T00:10:57.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:10:57.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:10:57.012 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:10:57.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:57.072 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:57.105 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-74 2023-09-30T00:10:57.121 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-74 2023-09-30T00:10:57.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:10:57.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:10:57.121 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:10:57.173 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:10:57.173 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:10:57.215 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:10:57.213+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:02.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:02.214+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:07.216 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:07.215+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:12.217+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:17.218 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:17.217+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:22.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:22.218+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:27.219 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:27.219+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:32.220 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:32.219+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:37.221 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:37.220+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:42.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:42.221+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:47.222 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:47.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:52.223 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:52.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:11:57.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:11:57.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:02.224 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:02.223+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:07.225 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:07.224+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:12.226 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:12.225+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:17.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:17.226+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:22.227 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:22.227+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:27.228 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:27.228+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:32.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:32.228+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:37.229 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:37.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:42.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:42.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:47.230 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:47.230+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:52.231 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:52.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:12:57.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:12:57.230+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:02.232 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:02.231+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:07.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:07.232+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:12.233 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:12.233+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:17.234 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:17.233+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:22.235 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:22.234+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:27.236 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:27.235+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:32.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:32.236+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:37.237 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:37.237+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:42.238 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:42.238+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:47.239 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:47.239+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:52.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:52.238+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:13:57.240 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:13:57.239+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:02.241 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:02.240+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:07.242 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:07.241+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:12.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:12.242+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:17.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:17.243+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:22.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:22.244+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:27.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:27.244+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:32.245 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:32.245+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:37.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:37.245+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:42.246 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:42.246+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:47.247 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:47.247+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:52.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:52.246+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:14:57.248 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:14:57.247+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:02.249 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:02.248+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:07.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:07.249+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:12.250 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:12.250+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:17.251 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:17.251+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:22.252 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:22.252+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:27.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:27.252+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:32.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:32.253+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:37.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:37.253+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:42.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:42.254+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:47.255 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:47.254+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:15:52.256 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:52.255+0000 7fee76610880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:15:52.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:52.255+0000 7fee76610880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:15:52.257 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:52.255+0000 7fee76610880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:15:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-30T00:15:52.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-09-30T00:15:52.280 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:52.280 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:52.281 DEBUG:boto:Method: PUT 2023-09-30T00:15:52.281 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:52.281 DEBUG:boto:Data: 2023-09-30T00:15:52.281 DEBUG:boto:Headers: {} 2023-09-30T00:15:52.281 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:52.281 DEBUG:boto:Port: 8000 2023-09-30T00:15:52.281 DEBUG:boto:Params: {} 2023-09-30T00:15:52.282 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:15:52.282 DEBUG:boto:Token: None 2023-09-30T00:15:52.282 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:15:52 GMT /dmuhsu-75/ 2023-09-30T00:15:52.282 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EDdS5YPQ2eskQUNiw9mSb0YIalI= 2023-09-30T00:15:52.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EDdS5YPQ2eskQUNiw9mSb0YIalI=', 'Content-Length': '0'} 2023-09-30T00:15:52.306 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047b41f88e63588bf-00651768b8-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:52.306 DEBUG:rgw_multi.tests:created bucket=dmuhsu-75 2023-09-30T00:15:52.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.307 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:15:52.666 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:52.666 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:52.717 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:15:52.736 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:15:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:15:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:15:52.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:15:52.302803Z" 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.737 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:15:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:15:52.738 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.738 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:15:52.739 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:52.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:15:52.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.739 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:15:52.792 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:52.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:52.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:15:52.825+0000 7f760af1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:52.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:15:52.826+0000 7f760af1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:52.840 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:15:52.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:15:52.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:15:52.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:15:52.844 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:15:52.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:15:52.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:15:52.847 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:15:52.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:15:52.847 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:15:52.847 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:15:52.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:15:52Z" } 2023-09-30T00:15:52.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:15:52.848 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:52.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:52.848 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:15:52.848 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:15:52.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:52.849 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:15:52.896 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:52.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:52.927+0000 7f959d810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:52.929 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:52.928+0000 7f959d810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:52.943 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:15:52.943 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:15:52.943 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:15:52.943 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:52.944 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:15:52.945 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.946 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032952.295269_705.1", 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:15:52.295269Z", 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.947 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:15:52.948 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:15:52.949 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:15:52.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.295269_705.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.295269Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:15:52Z" } 2023-09-30T00:15:52.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.295269_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.295269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:15:52.952 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:52.952 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.295269_705.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:52.952 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696032952.302803_708.1 target=1_1696032952.295269_705.1 2023-09-30T00:15:57.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:57.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:15:57.954 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:15:58.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:58.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:58.047 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:58.046+0000 7ff768267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:58.048 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:58.047+0000 7ff768267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:15:58.076 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:15:58.077 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:58.078 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:58.079 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:58.080 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:15:58.081 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.082 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:15:58.083 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:15:58.083 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:15:58.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:15:58.083 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:15:58Z" 2023-09-30T00:15:58.083 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:15:58.084 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:15:58Z" } 2023-09-30T00:15:58.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:15:58.084 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:58.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:15:58.085 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:15:58.085 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:58.085 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:58.085 DEBUG:boto:Method: HEAD 2023-09-30T00:15:58.085 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:58.085 DEBUG:boto:Data: 2023-09-30T00:15:58.085 DEBUG:boto:Headers: {} 2023-09-30T00:15:58.085 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.085 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.086 DEBUG:boto:Params: {} 2023-09-30T00:15:58.086 DEBUG:boto:Token: None 2023-09-30T00:15:58.086 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/ 2023-09-30T00:15:58.086 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:15:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:15:58.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b491e6525b0b91c1-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.089 DEBUG:boto:path=/dmuhsu-75/obj11 2023-09-30T00:15:58.089 DEBUG:boto:auth_path=/dmuhsu-75/obj11 2023-09-30T00:15:58.089 DEBUG:boto:Method: PUT 2023-09-30T00:15:58.089 DEBUG:boto:Path: /dmuhsu-75/obj11 2023-09-30T00:15:58.090 DEBUG:boto:Data: 2023-09-30T00:15:58.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:15:58.090 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.090 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.090 DEBUG:boto:Params: {} 2023-09-30T00:15:58.090 DEBUG:boto:Token: None 2023-09-30T00:15:58.090 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/obj11 2023-09-30T00:15:58.090 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NSF7wt0nmeKCeSQG552Y17l2GlQ= 2023-09-30T00:15:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NSF7wt0nmeKCeSQG552Y17l2GlQ='} 2023-09-30T00:15:58.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000379525ca63744858-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.097 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:58.098 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:58.098 DEBUG:boto:Method: HEAD 2023-09-30T00:15:58.098 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:58.098 DEBUG:boto:Data: 2023-09-30T00:15:58.098 DEBUG:boto:Headers: {} 2023-09-30T00:15:58.098 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.098 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.098 DEBUG:boto:Params: {} 2023-09-30T00:15:58.099 DEBUG:boto:Token: None 2023-09-30T00:15:58.099 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/ 2023-09-30T00:15:58.099 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:15:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:15:58.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b33962402bdfeb86-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.101 DEBUG:boto:path=/dmuhsu-75/obj22 2023-09-30T00:15:58.101 DEBUG:boto:auth_path=/dmuhsu-75/obj22 2023-09-30T00:15:58.101 DEBUG:boto:Method: PUT 2023-09-30T00:15:58.101 DEBUG:boto:Path: /dmuhsu-75/obj22 2023-09-30T00:15:58.101 DEBUG:boto:Data: 2023-09-30T00:15:58.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:15:58.102 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.102 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.102 DEBUG:boto:Params: {} 2023-09-30T00:15:58.102 DEBUG:boto:Token: None 2023-09-30T00:15:58.102 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/obj22 2023-09-30T00:15:58.102 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WGJ2PxzxGaJn9e2eTwKCg83tJLw= 2023-09-30T00:15:58.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WGJ2PxzxGaJn9e2eTwKCg83tJLw='} 2023-09-30T00:15:58.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e1d0cb0b1f8afb-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.106 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:58.107 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:58.107 DEBUG:boto:Method: HEAD 2023-09-30T00:15:58.107 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:58.107 DEBUG:boto:Data: 2023-09-30T00:15:58.107 DEBUG:boto:Headers: {} 2023-09-30T00:15:58.107 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.107 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.107 DEBUG:boto:Params: {} 2023-09-30T00:15:58.107 DEBUG:boto:Token: None 2023-09-30T00:15:58.108 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/ 2023-09-30T00:15:58.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:15:58.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:15:58.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057743ab241a3af66-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.110 DEBUG:boto:path=/dmuhsu-75/obj33 2023-09-30T00:15:58.110 DEBUG:boto:auth_path=/dmuhsu-75/obj33 2023-09-30T00:15:58.110 DEBUG:boto:Method: PUT 2023-09-30T00:15:58.110 DEBUG:boto:Path: /dmuhsu-75/obj33 2023-09-30T00:15:58.110 DEBUG:boto:Data: 2023-09-30T00:15:58.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:15:58.111 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.111 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.111 DEBUG:boto:Params: {} 2023-09-30T00:15:58.111 DEBUG:boto:Token: None 2023-09-30T00:15:58.111 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/obj33 2023-09-30T00:15:58.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0PBzzvuWax3BsfHbNIynADnIa4g= 2023-09-30T00:15:58.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0PBzzvuWax3BsfHbNIynADnIa4g='} 2023-09-30T00:15:58.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b5fd468f3678c44-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.117 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:58.117 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:58.117 DEBUG:boto:Method: HEAD 2023-09-30T00:15:58.117 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:58.117 DEBUG:boto:Data: 2023-09-30T00:15:58.117 DEBUG:boto:Headers: {} 2023-09-30T00:15:58.117 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.117 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.117 DEBUG:boto:Params: {} 2023-09-30T00:15:58.118 DEBUG:boto:Token: None 2023-09-30T00:15:58.118 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/ 2023-09-30T00:15:58.118 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:15:58.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:15:58.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000944c3338dc3f5d22-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.120 DEBUG:boto:path=/dmuhsu-75/obj44 2023-09-30T00:15:58.120 DEBUG:boto:auth_path=/dmuhsu-75/obj44 2023-09-30T00:15:58.120 DEBUG:boto:Method: PUT 2023-09-30T00:15:58.120 DEBUG:boto:Path: /dmuhsu-75/obj44 2023-09-30T00:15:58.121 DEBUG:boto:Data: 2023-09-30T00:15:58.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:15:58.121 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.121 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.121 DEBUG:boto:Params: {} 2023-09-30T00:15:58.121 DEBUG:boto:Token: None 2023-09-30T00:15:58.121 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/obj44 2023-09-30T00:15:58.121 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:g7Eg8beNycVmEF8ruF8NbOb+g+c= 2023-09-30T00:15:58.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g7Eg8beNycVmEF8ruF8NbOb+g+c='} 2023-09-30T00:15:58.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048815376a31914b8-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.171 DEBUG:boto:path=/dmuhsu-75/ 2023-09-30T00:15:58.171 DEBUG:boto:auth_path=/dmuhsu-75/ 2023-09-30T00:15:58.171 DEBUG:boto:Method: HEAD 2023-09-30T00:15:58.171 DEBUG:boto:Path: /dmuhsu-75/ 2023-09-30T00:15:58.171 DEBUG:boto:Data: 2023-09-30T00:15:58.171 DEBUG:boto:Headers: {} 2023-09-30T00:15:58.171 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.172 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.172 DEBUG:boto:Params: {} 2023-09-30T00:15:58.172 DEBUG:boto:Token: None 2023-09-30T00:15:58.172 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/ 2023-09-30T00:15:58.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:15:58.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:15:58.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064d42cb03834aa8a-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.175 DEBUG:boto:path=/dmuhsu-75/obj55 2023-09-30T00:15:58.175 DEBUG:boto:auth_path=/dmuhsu-75/obj55 2023-09-30T00:15:58.175 DEBUG:boto:Method: PUT 2023-09-30T00:15:58.175 DEBUG:boto:Path: /dmuhsu-75/obj55 2023-09-30T00:15:58.175 DEBUG:boto:Data: 2023-09-30T00:15:58.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:15:58.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:15:58.176 DEBUG:boto:Port: 8000 2023-09-30T00:15:58.176 DEBUG:boto:Params: {} 2023-09-30T00:15:58.176 DEBUG:boto:Token: None 2023-09-30T00:15:58.176 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 30 Sep 2023 00:15:58 GMT /dmuhsu-75/obj55 2023-09-30T00:15:58.176 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1yceiF7py6gt+AJIEfK+V2nbTK0= 2023-09-30T00:15:58.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1yceiF7py6gt+AJIEfK+V2nbTK0='} 2023-09-30T00:15:58.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d91f7979de6750c-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:15:58.182 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-75 2023-09-30T00:15:58.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:15:58.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:15:58.182 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:15:58.241 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:58.242 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:58.276 INFO:teuthology.orchestra.run.smithi184.stdout:Sync is disabled for bucket dmuhsu-75 2023-09-30T00:15:58.291 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-75 2023-09-30T00:15:58.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:15:58.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:15:58.292 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:15:58.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:15:58.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:15:58.385 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:15:58.383+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:16:03.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:03.385+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:16:08.386 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:08.386+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:16:13.387 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:13.387+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:16:18.389 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:18.388+0000 7f2eabf23880 1 bucket sync caught up with source: 2023-09-30T00:16:18.390 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.38.6, 00000000002.74.6, 00000000002.79.6] 2023-09-30T00:16:18.390 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.38.6, 00000000002.74.6, 00000000002.79.6] 2023-09-30T00:16:18.390 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:18.388+0000 7f2eabf23880 0 bucket checkpoint complete 2023-09-30T00:16:18.417 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-75 2023-09-30T00:16:18.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:16:18.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:16:18.418 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:16:18.474 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:16:18.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:16:18.537 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:18.537+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:23.538 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:23.538+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:28.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:28.538+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:33.539 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:33.539+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:38.540 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:38.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:43.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:43.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:48.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:48.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:53.541 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:53.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:16:58.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:16:58.541+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:03.542 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:03.542+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:08.543 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:08.543+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:13.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:13.543+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:18.544 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:18.544+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:23.545 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:23.544+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:28.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:28.545+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:33.546 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:33.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:38.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:38.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:43.547 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:43.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:48.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:48.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:53.548 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:53.547+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:17:58.549 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:17:58.548+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:03.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:03.549+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:08.550 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:08.550+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:13.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:13.550+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:18.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:18.551+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:23.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:23.551+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:28.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:28.552+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:33.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:33.552+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:38.553 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:38.553+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:43.554 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:43.553+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:48.555 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:48.554+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:53.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:53.554+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:18:58.556 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:18:58.556+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:03.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:03.556+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:08.557 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:08.557+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:13.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:13.557+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:18.558 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:18.558+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:23.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:23.558+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:28.559 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:28.559+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:33.560 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:33.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:38.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:38.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:43.561 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:43.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:48.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:48.561+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:53.562 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:53.561+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:19:58.563 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:19:58.563+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:03.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:03.563+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:08.564 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:08.564+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:13.565 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:13.564+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:18.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:18.565+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:23.566 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:23.566+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:28.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:28.566+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:33.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:33.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:38.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:38.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:43.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:43.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:48.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:48.568+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:53.569 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:53.568+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:20:58.570 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:20:58.569+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:21:03.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:03.570+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:21:08.571 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:08.571+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:21:13.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:13.571+0000 7f273308b880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:21:13.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:13.571+0000 7f273308b880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:21:13.572 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:13.571+0000 7f273308b880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:21:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-30T00:21:13.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-09-30T00:21:13.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.593 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:13.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:13.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:13.677 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:21:13.691 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:21:13.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:13.691 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:21:13.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:13.692 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:21:13.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:13.693 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:15:52.302803Z" 2023-09-30T00:21:13.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:13.694 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:21:13.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.694 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:21:13.696 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:13.696 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:21:13.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.697 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:13.751 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:13.751 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:13.784 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:13.783+0000 7f6a297a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:13.785 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:13.784+0000 7f6a297a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:13.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:13.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:13.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:13.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:13.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:13.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:13.805 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:13.806 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:13.807 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:13Z" 2023-09-30T00:21:13.808 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:13.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:13Z" } 2023-09-30T00:21:13.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:13.809 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:13.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:13.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:21:13.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:21:13.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:13.810 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:13.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:13.857 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:13.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:13.896+0000 7fba0a31f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:13.897 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:21:13.897+0000 7fba0a31f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:13.915 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:21:13.915 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:13.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:21:13.917 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.918 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:13.919 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.920 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:21:13.921 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.922 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:21:13.923 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:13.924 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:21:13Z" 2023-09-30T00:21:13.925 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:21:13.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:13Z" } 2023-09-30T00:21:13.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:13.927 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:13.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:13.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:21:13.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:13.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:13.928 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:13.970 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:13.970 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.035 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:21:14.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.036 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.120 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.121 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.172 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:21:14.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.173 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.218 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.219 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.263 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:14.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.264 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.307 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.307 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.371 INFO:teuthology.orchestra.run.smithi175.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-30T00:21:14.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.372 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.511 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:14.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.512 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.655 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:14.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.656 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.705 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.706 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.772 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:14.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.773 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.821 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.822 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.873 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:21:14.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.874 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:14.917 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:14.917 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:14.967 INFO:teuthology.orchestra.run.smithi175.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-09-30T00:21:14.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:14.968 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:15.051 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:15.051 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:15.113 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 9, 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.114 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.115 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:21:15.116 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.117 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:21:15.118 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.119 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:21:15.120 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.121 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:21:15.122 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:21:15.123 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.130 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:21:15.130 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: "directional": [ 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: "source_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: "dest_zone": "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.131 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.132 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:21:15.133 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.134 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:21:15.135 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:15.136 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:21:15.137 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:21:15.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:21:15.139 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:21:15.140 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:21:15.141 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:21:15.142 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-76 2023-09-30T00:21:15.142 DEBUG:boto:path=/dmuhsu-76/ 2023-09-30T00:21:15.142 DEBUG:boto:auth_path=/dmuhsu-76/ 2023-09-30T00:21:15.143 DEBUG:boto:Method: PUT 2023-09-30T00:21:15.143 DEBUG:boto:Path: /dmuhsu-76/ 2023-09-30T00:21:15.143 DEBUG:boto:Data: 2023-09-30T00:21:15.143 DEBUG:boto:Headers: {} 2023-09-30T00:21:15.143 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:21:15.143 DEBUG:boto:Port: 8000 2023-09-30T00:21:15.143 DEBUG:boto:Params: {} 2023-09-30T00:21:15.144 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:21:15.144 DEBUG:boto:Token: None 2023-09-30T00:21:15.144 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:21:15 GMT /dmuhsu-76/ 2023-09-30T00:21:15.144 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4SxQSY6Jp8ica0ER0zaKNPjO/ss= 2023-09-30T00:21:15.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:21:15 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4SxQSY6Jp8ica0ER0zaKNPjO/ss=', 'Content-Length': '0'} 2023-09-30T00:21:15.559 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000688e26f277070d16-00651769fb-21c9-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:21:15.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.560 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:15.629 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:15.630 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:15.779 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:21:15.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.780 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:15.848 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:15.848 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:15.973 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:15.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:15.973 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.025 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.097 INFO:teuthology.orchestra.run.smithi175.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-30T00:21:16.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.098 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.146 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.254 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:16.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.255 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.310 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.311 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.388 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:16.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.389 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.502 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:21:16.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.503 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.552 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.552 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.629 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:21:16.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:21:16.629 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:21:16.673 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.674 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.719 INFO:teuthology.orchestra.run.smithi175.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-09-30T00:21:16.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:16.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:16.721 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:16.764 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.764 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.801 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:21:16.819 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:21:16.819 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:16.819 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:16.819 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:21:16.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:16.820 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:21:16.821 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:21:16.606949Z" 2023-09-30T00:21:16.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:21:16.821 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:21:16.821 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:16.821 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:21:16.822 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:21:16.822 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:21:16.823 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:16.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:21:16.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:16.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:16.824 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:16.872 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:16.872 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:16.904 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:16.904+0000 7f73688b6880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:16.905 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:16.905+0000 7f73688b6880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:16.918 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:16.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:16.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.920 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:16.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:16.920 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:16.920 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:16.920 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:16.921 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:16.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:16.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:16.922 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:16.922 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:16.923 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:16.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:16.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:16.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:16.924 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:16.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:16.926 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:16.927 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:16.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:16.929 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:16.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:16.929 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:16Z" 2023-09-30T00:21:16.929 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:16.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:16Z" } 2023-09-30T00:21:16.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:16.930 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:16.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:16.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:21.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:21.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:21.933 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:21.985 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:21.985 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:22.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:22.018+0000 7f2002014880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:22.019 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:22.019+0000 7f2002014880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:22.033 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:22.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:22.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:22Z" 2023-09-30T00:21:22.039 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:22.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:22Z" } 2023-09-30T00:21:22.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:22.041 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:22.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:22.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:27.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:27.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:27.042 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:27.095 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:27.098 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:27.131 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:27.130+0000 7efc6d127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:27.131 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:27.131+0000 7efc6d127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:27.147 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:27.147 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:27.147 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:27.148 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:27.149 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:27.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:27.151 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:27.152 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:27.153 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:27.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:27.154 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:27Z" 2023-09-30T00:21:27.154 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:27.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:27Z" } 2023-09-30T00:21:27.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:27.155 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:27.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:27.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:32.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:32.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:32.157 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:32.210 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:32.210 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:32.243 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:32.243+0000 7f9a7951e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:32.244 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:32.244+0000 7f9a7951e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:32.259 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:32.259 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:32.259 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:32.259 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:32.260 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:32.261 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:32.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:32.263 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:32.264 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:32.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:32.266 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:32.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:32.266 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:32Z" 2023-09-30T00:21:32.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:32.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:32Z" } 2023-09-30T00:21:32.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:32.267 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:32.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:32.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:37.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:37.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:37.269 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:37.321 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:37.322 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:37.355 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:37.354+0000 7fe00a1af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:37.356 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:37.355+0000 7fe00a1af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:37.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:37Z" 2023-09-30T00:21:37.377 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:37.378 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:37Z" } 2023-09-30T00:21:37.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:37.379 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:37.379 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:37.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:42.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:42.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:42.380 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:42.433 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:42.433 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:42.466 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:42.465+0000 7f3a20f4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:42.467 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:42.466+0000 7f3a20f4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:42.481 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:42.481 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:42.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:42.483 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:42.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:42.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:42.486 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:42.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:42.488 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:42Z" 2023-09-30T00:21:42.488 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:42.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:42Z" } 2023-09-30T00:21:42.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:42.489 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:42.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:42.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:47.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:47.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:47.490 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:47.544 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:47.544 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:47.577 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:47.577+0000 7f8cc6926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:47.578 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:47.578+0000 7f8cc6926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:47.592 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.593 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:47.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:47.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:47.596 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:47.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:47.598 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:47Z" 2023-09-30T00:21:47.599 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:47.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:47Z" } 2023-09-30T00:21:47.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:47.601 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:47.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:47.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:52.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:52.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:52.602 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:52.655 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:52.655 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:52.688 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:52.688+0000 7f0760d1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:52.689 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:52.689+0000 7f0760d1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:52.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:52.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:52.706 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:52.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:52.708 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:52.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:52.710 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:52Z" 2023-09-30T00:21:52.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:52.711 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:52Z" } 2023-09-30T00:21:52.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:52.712 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:52.712 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:52.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:21:57.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:57.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:21:57.714 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:21:57.767 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:21:57.767 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:21:57.800 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:57.800+0000 7f399692a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:57.801 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:21:57.801+0000 7f399692a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:21:57.816 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:21:57.816 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:21:57.816 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:21:57.816 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:21:57.816 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:57.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:57.818 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:57.819 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:57.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:21:57.821 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:57.821 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:57.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:57.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:21:57.821 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:21:57.822 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:21:57.823 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:21:57.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:21:57.824 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:21:57Z" 2023-09-30T00:21:57.824 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:21:57.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:21:57Z" } 2023-09-30T00:21:57.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:21:57.825 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:57.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:21:57.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:02.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:02.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:02.827 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:02.879 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:02.879 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:02.913 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:02.912+0000 7fd16e524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:02.914 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:02.913+0000 7fd16e524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:02.928 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.929 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:02.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:02.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:02.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:02.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:02.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:02.935 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:02.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:02.936 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:02Z" 2023-09-30T00:22:02.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:02.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:02Z" } 2023-09-30T00:22:02.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:02.937 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:02.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:02.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:07.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:07.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:07.939 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:07.992 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:07.992 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:08.025 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:08.025+0000 7f20b0810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:08.026 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:08.026+0000 7f20b0810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:08.041 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:08.042 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:08.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:08.044 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:08.045 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:08.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:08Z" 2023-09-30T00:22:08.047 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:08.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:08Z" } 2023-09-30T00:22:08.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:08.049 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:08.049 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:08.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:13.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:13.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:13.050 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:13.105 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:13.105 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:13.144 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:13.143+0000 7f59a6921880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:13.145 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:13.144+0000 7f59a6921880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.160 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:13.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:13.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:13.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:13.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:13.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:13Z" 2023-09-30T00:22:13.167 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:13.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:13Z" } 2023-09-30T00:22:13.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:13.169 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:13.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:13.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:18.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:18.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:18.170 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:18.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:18.258 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:18.257+0000 7fe47340a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:18.259 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:18.258+0000 7fe47340a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:18.274 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:18.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:18.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:18.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:18.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:18.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:18.281 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:18Z" 2023-09-30T00:22:18.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:18.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:18Z" } 2023-09-30T00:22:18.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:18.282 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:18.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:18.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:23.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:23.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:23.284 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:23.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:23.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:23.370 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:23.370+0000 7f3aae727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:23.371 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:23.371+0000 7f3aae727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:23.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:23.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:23.393 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:23Z" 2023-09-30T00:22:23.393 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:23.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:23Z" } 2023-09-30T00:22:23.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:23.394 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:23.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:23.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:28.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:28.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:28.396 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:28.449 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:28.449 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:28.482 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:28.481+0000 7fa25fb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:28.483 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:28.482+0000 7fa25fb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:28Z" 2023-09-30T00:22:28.504 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:28.505 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:28Z" } 2023-09-30T00:22:28.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:28.505 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:28.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:28.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:33.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:33.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:33.507 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:33.560 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:33.560 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:33.593 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:33.593+0000 7f2c17213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:33.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:33.594+0000 7f2c17213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:33.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.610 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:33.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:33.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:33.613 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:33.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:33.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:33Z" 2023-09-30T00:22:33.616 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:33.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:33Z" } 2023-09-30T00:22:33.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:33.618 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:33.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:33.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:38.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:38.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:38.620 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:38.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:38.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:38.705 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:38.705+0000 7f991c48b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:38.706 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:38.705+0000 7f991c48b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.721 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:38.722 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:38.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:38.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:38.725 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.726 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:38.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:38.727 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:38.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:38.727 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:38Z" 2023-09-30T00:22:38.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:38.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:38Z" } 2023-09-30T00:22:38.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:38.728 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:38.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:38.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:43.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:43.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:43.730 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:43.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:43.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:43.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:43.816+0000 7f9345122880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:43.818 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:43.817+0000 7f9345122880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:43.833 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:43.833 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:43.834 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:43.835 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:43.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:43.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:43Z" 2023-09-30T00:22:43.839 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:43.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:43Z" } 2023-09-30T00:22:43.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:43.841 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:43.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:43.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:48.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:48.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:48.843 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:48.930 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:48.929+0000 7fa43be0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:48.931 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:48.930+0000 7fa43be0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:48.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:48.946 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:48.947 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:48.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:48.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:48.950 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:48.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:48.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:48.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:48.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:48.952 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:48.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:48.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:48.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:48.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:48.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:48.955 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:48.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:48.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:48.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:48.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:48.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:48.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:48.957 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:48.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.957 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:48.958 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:48.958 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:48.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:48.958 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:48Z" 2023-09-30T00:22:48.958 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:48.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:48Z" } 2023-09-30T00:22:48.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:48.961 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:48.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:48.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:53.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:53.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:53.963 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:54.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:54.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:54.049 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:54.049+0000 7f200780a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:54.050 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:54.050+0000 7f200780a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:54.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:54.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:54.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:54.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:54.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:54.070 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:54Z" 2023-09-30T00:22:54.071 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:54.072 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:54Z" } 2023-09-30T00:22:54.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:54.072 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:54.072 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:54.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:22:59.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:59.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:22:59.074 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:22:59.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:22:59.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:22:59.162 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:59.162+0000 7fb906b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:59.163 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:22:59.163+0000 7fb906b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:22:59.178 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:22:59.178 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:22:59.178 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:22:59.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:22:59.178 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:59.179 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:22:59.180 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:59.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:22:59.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:22:59Z" 2023-09-30T00:22:59.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:22:59.185 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:22:59Z" } 2023-09-30T00:22:59.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:22:59.186 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:59.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:22:59.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:23:04.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:04.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:04.187 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:04.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:04.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:04.277 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:04.277+0000 7fcae85af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:04.278 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:04.278+0000 7fcae85af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:04.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:04.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:04.295 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:04.296 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:04.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:04Z" 2023-09-30T00:23:04.299 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:04.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696032952.302803_708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:15:52.302803Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:04Z" } 2023-09-30T00:23:04.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:04.301 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:04.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:04.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033276.606949_760.1 target=1_1696032952.302803_708.1 2023-09-30T00:23:09.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:09.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:09.303 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:09.354 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.354 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.387 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:09.386+0000 7f285c210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:09.388 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:09.387+0000 7f285c210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:09.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:09.402 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:09.402 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:09.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:09.403 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:09.404 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:09.405 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:09.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:09.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:09Z" 2023-09-30T00:23:09.408 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:09.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:09Z" } 2023-09-30T00:23:09.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:09.410 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:09.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:09.410 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:23:09.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:23:09.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:09.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:09.410 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:09.466 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.466 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.510 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:09.509+0000 7f8416b29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:09.511 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:09.510+0000 7f8416b29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:09.526 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:09.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:23:09.528 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:09.529 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.530 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:09.531 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:09.532 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:23:09.533 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:23:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:09.534 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:23:09Z" 2023-09-30T00:23:09.534 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:23:09.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:09Z" } 2023-09-30T00:23:09.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:09.535 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:09.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:09.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:23:09.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.536 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:09.583 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.584 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.634 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:09.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.635 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.686 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.759 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:09.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.759 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:09.810 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.810 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.877 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:09.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.878 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:09.923 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:09.923 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:09.975 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:23:09.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dmuhsu-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:09.976 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dmuhsu-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:10.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:10.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:10.068 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:23:10.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:10.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:10.069 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:10.152 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:10.153 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:10.197 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:23:10.216 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:23:10.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:10.216 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:10.216 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:10.217 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:23:09.961350Z" 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:10.218 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:23:10.219 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:10.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:23:10.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:10.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:10.220 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:10.272 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:10.272 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:10.306 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:10.305+0000 7fcb14e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:10.307 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:10.306+0000 7fcb14e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:10.322 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:10.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:10.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:10.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:10.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:10.327 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:10.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:10.329 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:10Z" 2023-09-30T00:23:10.329 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:10.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:10Z" } 2023-09-30T00:23:10.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:10.330 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:10.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:10.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033389.961350_768.1 target=1_1696033276.606949_760.1 2023-09-30T00:23:15.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:15.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:15.332 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:15.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:15.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:15.418 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:15.417+0000 7f0e0a722880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:15.418 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:15.418+0000 7f0e0a722880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:15.434 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:15.434 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:15.434 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:15.434 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:15.434 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:15.435 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:15.436 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:15.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:15.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:15.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:15.440 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:15Z" 2023-09-30T00:23:15.441 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:15.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:15Z" } 2023-09-30T00:23:15.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:15.442 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:15.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:15.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033389.961350_768.1 target=1_1696033276.606949_760.1 2023-09-30T00:23:20.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:20.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:20.444 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:20.502 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:20.502 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:20.550 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:20.549+0000 7f4646030880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:20.551 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:20.550+0000 7f4646030880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:20.566 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:20.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:20.568 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:20.569 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:20.570 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:20.571 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.572 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:20.573 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:20.573 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:20.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:20.573 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:20Z" 2023-09-30T00:23:20.573 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:20.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:20Z" } 2023-09-30T00:23:20.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:20.574 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:20.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:20.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033389.961350_768.1 target=1_1696033276.606949_760.1 2023-09-30T00:23:25.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:25.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:25.576 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:25.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:25.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:25.662 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:25.662+0000 7f8a75e13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:25.663 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:25.663+0000 7f8a75e13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:25.678 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:25.678 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:25.679 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:25.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:25.681 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033276.606949_760.1", 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:25.682 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:21:16.606949Z", 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:25.683 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:25.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:25Z" 2023-09-30T00:23:25.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:25.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033276.606949_760.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:16.606949Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:25Z" } 2023-09-30T00:23:25.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033276.606949_760.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:16.606949Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:25.686 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:25.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033276.606949_760.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:25.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033389.961350_768.1 target=1_1696033276.606949_760.1 2023-09-30T00:23:30.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:30.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:30.688 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:30.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:30.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:30.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:30.774+0000 7f20e0f29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:30.775 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:30.775+0000 7f20e0f29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:30Z" 2023-09-30T00:23:30.798 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:30.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:30Z" } 2023-09-30T00:23:30.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:30.800 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:30.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:30.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:23:30.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:23:30.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:30.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:30.801 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:30.853 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:30.853 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:30.889 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:30.889+0000 7f6c8092c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:30.890 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:30.890+0000 7f6c8092c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:30.905 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:23:30.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:30.907 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:30.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:30.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.910 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:23:30.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:23:30.911 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:23:30.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:30.911 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:23:30Z" 2023-09-30T00:23:30.911 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:23:30.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:30Z" } 2023-09-30T00:23:30.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:30.912 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:30.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:30.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:23:30.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:30.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:30.913 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:30.960 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:30.960 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.041 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:31.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.042 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:31.091 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.091 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.166 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:31.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.167 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:31.215 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.216 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.270 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:23:31.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.271 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:31.314 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.314 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.369 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:23:31.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.369 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:31.451 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.451 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.521 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:23:31.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:31.522 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.605 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 10, 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:23:31.670 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:23:31.671 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:23:31.672 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:31.673 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.674 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:31.675 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.676 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.678 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.680 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.681 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:31.682 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:23:31.683 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:23:31.684 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:31.685 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:31.686 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:31.687 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:23:31.689 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-09-30T00:23:31.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:31.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:31.690 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:31.739 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.739 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.809 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:23:31.843 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:23:31.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.843 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:23:09.961350Z" 2023-09-30T00:23:31.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:31.845 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:31.845 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:31.845 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:23:31.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:31.845 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:23:31.846 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:31.846 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:23:31.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:31.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:31.847 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:31.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:31.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:31.994 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:31.993+0000 7f5d86132880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:31.996 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:31.995+0000 7f5d86132880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:32.031 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:32.032 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:32.033 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:32.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:32.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:31Z" 2023-09-30T00:23:32.036 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:32.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:31Z" } 2023-09-30T00:23:32.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:32.038 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:32.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:32.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:23:32.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:23:32.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:32.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:32.039 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:32.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:32.119 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:32.253 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:32.253+0000 7f7414faf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:32.254 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:23:32.254+0000 7f7414faf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:32.323 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:23:32.323 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:23:32.323 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:23:32.323 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:23:32.323 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:32.324 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:23:32Z" 2023-09-30T00:23:32.329 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:23:32.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:32Z" } 2023-09-30T00:23:32.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:32.331 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:32.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:32.331 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:23:32.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.331 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:32.392 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:32.392 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:32.523 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:23:32.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.524 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:32.577 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:32.577 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:32.653 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:23:32.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.654 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:32.709 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:32.709 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:32.826 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:23:32.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.826 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:32.878 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:32.878 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:32.970 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:23:32.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:32.971 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:33.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:33.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:33.089 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:23:33.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:33.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:33.089 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:33.134 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:33.134 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:33.186 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:23:33.186 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:33.186 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 11, 2023-09-30T00:23:33.186 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:23:33.186 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:23:33.187 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:23:33.188 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:33.189 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:33.190 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:23:33.191 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:33.192 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.193 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:23:33.194 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:23:33.195 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:23:33.196 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:23:33.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:23:33.198 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.202 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.203 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:23:33.204 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:23:33.205 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.206 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.207 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.208 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:23:33.209 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:23:33.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:23:33.211 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:33.212 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:23:33.213 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:23:33.214 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:23:33.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:33.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:23:33.216 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:23:33.258 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:33.258 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:33.349 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:23:33.350 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-77 2023-09-30T00:23:33.350 DEBUG:boto:path=/dmuhsu-77/ 2023-09-30T00:23:33.350 DEBUG:boto:auth_path=/dmuhsu-77/ 2023-09-30T00:23:33.350 DEBUG:boto:Method: PUT 2023-09-30T00:23:33.350 DEBUG:boto:Path: /dmuhsu-77/ 2023-09-30T00:23:33.351 DEBUG:boto:Data: 2023-09-30T00:23:33.351 DEBUG:boto:Headers: {} 2023-09-30T00:23:33.351 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:23:33.351 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.351 DEBUG:boto:Params: {} 2023-09-30T00:23:33.352 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:23:33.352 DEBUG:boto:Token: None 2023-09-30T00:23:33.352 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-77/ 2023-09-30T00:23:33.353 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+meUkowXHhFbY0Ve2o73rRNDblg= 2023-09-30T00:23:33.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+meUkowXHhFbY0Ve2o73rRNDblg=', 'Content-Length': '0'} 2023-09-30T00:23:33.545 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004887aad963a6005d-0065176a85-22d4-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.545 DEBUG:boto:path=/dmuhsu-77/ 2023-09-30T00:23:33.545 DEBUG:boto:auth_path=/dmuhsu-77/ 2023-09-30T00:23:33.545 DEBUG:boto:Method: HEAD 2023-09-30T00:23:33.546 DEBUG:boto:Path: /dmuhsu-77/ 2023-09-30T00:23:33.546 DEBUG:boto:Data: 2023-09-30T00:23:33.546 DEBUG:boto:Headers: {} 2023-09-30T00:23:33.546 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:23:33.546 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.546 DEBUG:boto:Params: {} 2023-09-30T00:23:33.546 DEBUG:boto:Token: None 2023-09-30T00:23:33.546 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-77/ 2023-09-30T00:23:33.547 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:RWDezcbD/wBb3iEyPwORexPWKwQ= 2023-09-30T00:23:33.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RWDezcbD/wBb3iEyPwORexPWKwQ=', 'Content-Length': '0'} 2023-09-30T00:23:33.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e14625de03b4c25-0065176a85-22d4-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.550 DEBUG:boto:path=/dmuhsu-77/obj1 2023-09-30T00:23:33.550 DEBUG:boto:auth_path=/dmuhsu-77/obj1 2023-09-30T00:23:33.550 DEBUG:boto:Method: PUT 2023-09-30T00:23:33.550 DEBUG:boto:Path: /dmuhsu-77/obj1 2023-09-30T00:23:33.551 DEBUG:boto:Data: 2023-09-30T00:23:33.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:23:33.551 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:23:33.551 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.551 DEBUG:boto:Params: {} 2023-09-30T00:23:33.551 DEBUG:boto:Token: None 2023-09-30T00:23:33.552 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-77/obj1 2023-09-30T00:23:33.552 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v9lTnJGaOeiLwJOEO85BcMKiapE= 2023-09-30T00:23:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v9lTnJGaOeiLwJOEO85BcMKiapE='} 2023-09-30T00:23:33.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000789b2270bf798b9e-0065176a85-22d4-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.584 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-78 2023-09-30T00:23:33.584 DEBUG:boto:path=/dmuhsu-78/ 2023-09-30T00:23:33.585 DEBUG:boto:auth_path=/dmuhsu-78/ 2023-09-30T00:23:33.585 DEBUG:boto:Method: PUT 2023-09-30T00:23:33.585 DEBUG:boto:Path: /dmuhsu-78/ 2023-09-30T00:23:33.585 DEBUG:boto:Data: 2023-09-30T00:23:33.585 DEBUG:boto:Headers: {} 2023-09-30T00:23:33.585 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:23:33.586 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.586 DEBUG:boto:Params: {} 2023-09-30T00:23:33.590 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:23:33.590 DEBUG:boto:Token: None 2023-09-30T00:23:33.590 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-78/ 2023-09-30T00:23:33.591 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qyPqTn7/+htzE2NtL4HDWeLyVEs= 2023-09-30T00:23:33.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qyPqTn7/+htzE2NtL4HDWeLyVEs=', 'Content-Length': '0'} 2023-09-30T00:23:33.655 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021b03b5c6b51e271-0065176a85-1a63-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.655 DEBUG:boto:path=/dmuhsu-78/ 2023-09-30T00:23:33.656 DEBUG:boto:auth_path=/dmuhsu-78/ 2023-09-30T00:23:33.656 DEBUG:boto:Method: HEAD 2023-09-30T00:23:33.656 DEBUG:boto:Path: /dmuhsu-78/ 2023-09-30T00:23:33.656 DEBUG:boto:Data: 2023-09-30T00:23:33.656 DEBUG:boto:Headers: {} 2023-09-30T00:23:33.656 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:23:33.656 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.656 DEBUG:boto:Params: {} 2023-09-30T00:23:33.657 DEBUG:boto:Token: None 2023-09-30T00:23:33.657 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-78/ 2023-09-30T00:23:33.657 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:okjWBywfwNDIFXLWbX32Qc6bnoA= 2023-09-30T00:23:33.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:okjWBywfwNDIFXLWbX32Qc6bnoA=', 'Content-Length': '0'} 2023-09-30T00:23:33.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000758c40943f98693e-0065176a85-1a63-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.659 DEBUG:boto:path=/dmuhsu-78/obj2 2023-09-30T00:23:33.659 DEBUG:boto:auth_path=/dmuhsu-78/obj2 2023-09-30T00:23:33.660 DEBUG:boto:Method: PUT 2023-09-30T00:23:33.660 DEBUG:boto:Path: /dmuhsu-78/obj2 2023-09-30T00:23:33.660 DEBUG:boto:Data: 2023-09-30T00:23:33.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:23:33.660 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:23:33.660 DEBUG:boto:Port: 8000 2023-09-30T00:23:33.660 DEBUG:boto:Params: {} 2023-09-30T00:23:33.660 DEBUG:boto:Token: None 2023-09-30T00:23:33.660 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:23:33 GMT /dmuhsu-78/obj2 2023-09-30T00:23:33.661 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MBTVh4wCvyf1qTP0+HWWtBuwpas= 2023-09-30T00:23:33.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:23:33 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MBTVh4wCvyf1qTP0+HWWtBuwpas='} 2023-09-30T00:23:33.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c043becabe3a1bbc-0065176a85-1a63-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:23:33.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:33.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:33.669 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:33.731 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:33.732 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:33.827 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:23:33.638104Z" 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:23:33.872 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:23:33.873 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:23:33.873 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:33.874 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:23:33.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:33.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:33.874 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:33.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:33.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:33.949 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:33.949+0000 7fcc1154b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:33.950 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:33.949+0000 7fcc1154b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:33.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:33.963 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:33.963 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:33.964 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:33.965 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:33.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:33.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:33Z" 2023-09-30T00:23:33.969 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:33.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:33Z" } 2023-09-30T00:23:33.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:33.971 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:33.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:33.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:23:38.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:38.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:38.972 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:39.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:39.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:39.059 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:39.058+0000 7f09c6212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:39.060 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:39.059+0000 7f09c6212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:39.074 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:39.074 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:39.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:39.076 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:39.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:39.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:39.079 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:39Z" 2023-09-30T00:23:39.080 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:39.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:39Z" } 2023-09-30T00:23:39.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:39.082 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:39.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:39.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:23:44.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:44.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:44.083 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:44.136 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:44.136 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:44.169 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:44.169+0000 7f62d9a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:44.170 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:44.170+0000 7f62d9a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:44.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:44.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:44.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:44.190 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.191 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:44.192 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:44.192 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:44.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:44.192 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:44Z" 2023-09-30T00:23:44.192 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:44.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:44Z" } 2023-09-30T00:23:44.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:44.193 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:44.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:44.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:23:49.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:49.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:49.195 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:49.248 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:49.248 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:49.282 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:49.281+0000 7f46a831e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:49.282 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:49.282+0000 7f46a831e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:49.297 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:49.297 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:49.298 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:49.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:49.300 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:49.301 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:49.302 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:49.303 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:49.304 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:49.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:49.304 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:49Z" 2023-09-30T00:23:49.304 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:49.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:49Z" } 2023-09-30T00:23:49.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:49.305 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:49.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:49.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:23:54.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:54.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:54.306 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:54.359 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:54.359 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:54.393 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:54.392+0000 7f296dc8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:54.394 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:54.393+0000 7f296dc8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:54.408 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:54.408 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:54.409 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:54.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:54.411 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:54.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:54.413 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:54.414 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:54.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:54.415 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:54Z" 2023-09-30T00:23:54.415 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:54.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:54Z" } 2023-09-30T00:23:54.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:54.416 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:54.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:54.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:23:59.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:59.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:23:59.417 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:23:59.470 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:23:59.470 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:23:59.504 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:59.503+0000 7f202f815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:59.504 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:23:59.504+0000 7f202f815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:23:59.519 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:23:59.519 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:23:59.519 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:23:59.519 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:23:59.519 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:23:59.520 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:59.521 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:59.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:23:59.523 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:23:59.524 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:23:59.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:23:59.526 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:23:59Z" 2023-09-30T00:23:59.526 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:23:59.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:23:59Z" } 2023-09-30T00:23:59.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:23:59.527 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:59.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:23:59.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:04.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:04.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:04.529 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:04.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:04.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:04.616 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:04.615+0000 7fdb43ddc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:04.616 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:04.616+0000 7fdb43ddc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:04.631 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:04.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:04.633 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:04.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:04.636 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:04.637 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:04Z" 2023-09-30T00:24:04.638 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:04.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:04Z" } 2023-09-30T00:24:04.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:04.639 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:04.639 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:04.639 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:09.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:09.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:09.641 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:09.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:09.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:09.728 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:09.727+0000 7f67aa615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:09.729 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:09.728+0000 7f67aa615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:09.743 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:09.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:09.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:09.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:09.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:09.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:09.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:09Z" 2023-09-30T00:24:09.751 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:09.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:09Z" } 2023-09-30T00:24:09.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:09.752 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:09.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:09.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:14.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:14.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:14.754 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:14.807 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:14.807 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:14.840 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:14.840+0000 7f38b1212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:14.841 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:14.840+0000 7f38b1212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:14.856 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:14.856 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:14.856 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:14.857 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:14.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:14Z" 2023-09-30T00:24:14.863 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:14.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:14Z" } 2023-09-30T00:24:14.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:14.864 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:14.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:14.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:19.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:19.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:19.866 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:19.919 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:19.919 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:19.952 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:19.952+0000 7ff5add4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:19.953 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:19.953+0000 7ff5add4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:19.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:19.968 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:19.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:19.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:19.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:19.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:19.970 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:19.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:19.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:19.973 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:19Z" 2023-09-30T00:24:19.974 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:19.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:19Z" } 2023-09-30T00:24:19.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:19.975 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:19.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:19.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:24.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:24.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:24.977 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:25.030 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:25.030 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:25.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:25.063+0000 7fb62172a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:25.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:25.064+0000 7fb62172a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:25.079 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:25.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:25.081 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:25.082 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:25.084 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.085 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:25.086 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:25.086 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:25.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:25.086 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:25Z" 2023-09-30T00:24:25.086 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:25.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:25Z" } 2023-09-30T00:24:25.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:25.087 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:25.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:25.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:30.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:30.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:30.089 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:30.141 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:30.141 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:30.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:30.175+0000 7f7cb4529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:30.176 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:30.176+0000 7f7cb4529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:30.190 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:30.190 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:30.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:30.193 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:30.193 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:30.193 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:30.193 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:30.194 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:30.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:30.197 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:30.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:30.199 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:30Z" 2023-09-30T00:24:30.199 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:30.200 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:30Z" } 2023-09-30T00:24:30.200 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:30.200 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:30.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:30.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:35.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:35.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:35.202 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:35.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:35.255 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:35.288 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:35.287+0000 7f5da272d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:35.289 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:35.288+0000 7f5da272d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:35.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:35.308 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:35.309 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:35Z" 2023-09-30T00:24:35.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:35.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:35Z" } 2023-09-30T00:24:35.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:35.311 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:35.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:35.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:40.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:40.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:40.312 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:40.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:40.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:40.398 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:40.397+0000 7f061b010880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:40.399 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:40.398+0000 7f061b010880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:40.413 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:40.413 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:40.413 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:40.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:40.413 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:40.414 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:40.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:40.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:40.418 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:40.419 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:40Z" 2023-09-30T00:24:40.420 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:40.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:40Z" } 2023-09-30T00:24:40.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:40.421 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:40.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:40.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:45.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:45.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:45.423 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:45.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:45.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:45.511 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:45.511+0000 7f5334819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:45.512 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:45.512+0000 7f5334819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:45.528 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:45.528 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:45.529 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:45.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:45.531 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:45.533 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:45.534 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:45.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:45.535 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:45Z" 2023-09-30T00:24:45.535 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:45.537 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:45Z" } 2023-09-30T00:24:45.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:45.538 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:45.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:45.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:50.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:50.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:50.540 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:50.594 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:50.594 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:50.627 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:50.626+0000 7f2dc9e10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:50.627 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:50.627+0000 7f2dc9e10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:50.642 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:50Z" 2023-09-30T00:24:50.649 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:50.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:50Z" } 2023-09-30T00:24:50.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:50.650 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:50.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:50.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:24:55.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:55.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:24:55.651 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:24:55.705 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:24:55.705 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:24:55.738 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:55.737+0000 7f2b8c736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:55.739 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:24:55.738+0000 7f2b8c736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:24:55.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:55.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:55.756 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:55.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:24:55.759 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:24:55.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:24:55.761 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:24:55Z" 2023-09-30T00:24:55.761 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:24:55.762 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:24:55Z" } 2023-09-30T00:24:55.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:24:55.762 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:55.762 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:24:55.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:25:00.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:00.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:00.764 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:00.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:00.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:00.850 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:00.849+0000 7f576fd2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:00.850 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:00.850+0000 7f576fd2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:00.865 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.866 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:00Z" 2023-09-30T00:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:00.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:00Z" } 2023-09-30T00:25:00.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:00.873 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:00.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:00.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:25:05.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:05.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:05.875 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:05.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:05.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:05.962 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:05.961+0000 7ff78cac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:05.962 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:05.962+0000 7ff78cac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:05.977 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.978 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:05.979 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:05.980 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:05.981 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:05.982 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:05.983 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:05Z" 2023-09-30T00:25:05.984 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:05.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:05Z" } 2023-09-30T00:25:05.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:05.986 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:05.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:05.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:25:10.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:10.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:10.987 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:11.040 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:11.041 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:11.075 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:11.074+0000 7fa8ec733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:11.076 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:11.075+0000 7fa8ec733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:11.090 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.091 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:11.092 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:11.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:11.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033389.961350_768.1", 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:09.961350Z", 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:11.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:11Z" 2023-09-30T00:25:11.097 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:11.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033389.961350_768.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:09.961350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:11Z" } 2023-09-30T00:25:11.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033389.961350_768.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:09.961350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:11.099 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:11.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033389.961350_768.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:11.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033389.961350_768.1 2023-09-30T00:25:16.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:16.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:16.101 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:16.155 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:16.156 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:16.190 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:16.189+0000 7fbfaf737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:16.190 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:16.190+0000 7fbfaf737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:16.205 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:16.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:16.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:16.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:16.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:16.211 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:16.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:16.212 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:16Z" 2023-09-30T00:25:16.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:16.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:16Z" } 2023-09-30T00:25:16.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:16.213 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:16.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:16.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:25:16.213 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:25:16.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:16.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:16.214 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:16.267 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:16.268 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:16.304+0000 7f4123611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:16.305 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:16.305+0000 7f4123611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:16.320 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.321 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:16.322 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:16.323 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:25:16.324 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.494209_811.1", 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:33.494209Z", 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:16.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:25:16Z" 2023-09-30T00:25:16.327 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:25:16.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.494209_811.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.494209Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:16Z" } 2023-09-30T00:25:16.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.494209_811.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.494209Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:16.329 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:16.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.494209_811.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:16.329 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033413.638104_826.1 target=1_1696033413.494209_811.1 2023-09-30T00:25:21.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.330 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:21.383 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:21.384 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:21.422 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:21.422+0000 7f4cccb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:21.423 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:21.423+0000 7f4cccb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:21.438 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:25:21.438 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.439 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:25:21.441 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.441 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:25:21.441 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:21.441 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:21.441 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:21.443 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:25:21.444 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.445 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:21.446 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:25:21Z" 2023-09-30T00:25:21.447 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:25:21.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:21Z" } 2023-09-30T00:25:21.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:21.449 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:21.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:21.449 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:25:21.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.449 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:21.496 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:21.496 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:21.573 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000022827", 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:23:33.578629Z" 2023-09-30T00:25:21.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:21.575 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:25:21.576 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:21.576 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:25:21.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:21.577 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:21.632 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:21.632 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:21.681 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.682 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:25:21.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022686", 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:21.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:58.127982Z" 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:21.685 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:21.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:25:21.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:21.688 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:21Z" 2023-09-30T00:25:21.688 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:21.688 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022686", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:15:58.127982Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:25:21Z" } 2023-09-30T00:25:21.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022686', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:15:58.127982Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:25:21.689 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:21.689 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022686', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:21.689 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000022827 target=00000000000000000000:00000000000000022686 2023-09-30T00:25:26.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:26.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:26.691 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:26.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:26.746 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:26.796 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:26.796 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:26.797 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:26.798 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022686", 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:58.127982Z" 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:26.799 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:26.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.801 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:26.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:25:26.802 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:25:26.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:26.802 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:26Z" 2023-09-30T00:25:26.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:26.803 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022686", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:15:58.127982Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:25:26Z" } 2023-09-30T00:25:26.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022686', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:15:58.127982Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:25:26.803 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:26.803 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022686', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:26.803 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000022827 target=00000000000000000000:00000000000000022686 2023-09-30T00:25:31.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:31.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:31.805 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:31.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:31.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:31.913 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:31.913 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:31.913 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:31.913 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:31.914 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:31.915 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022686", 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:15:58.127982Z" 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:31.916 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:31.917 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:31.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:31Z" 2023-09-30T00:25:31.919 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:31.920 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022686", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:15:58.127982Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:25:31Z" } 2023-09-30T00:25:31.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022686', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:15:58.127982Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:25:31.920 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:31.921 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022686', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:31.921 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000022827 target=00000000000000000000:00000000000000022686 2023-09-30T00:25:36.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:36.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:36.922 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:36.977 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:36.977 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.028 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:25:37.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022827", 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.578629Z" 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:37.032 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:37Z" 2023-09-30T00:25:37.033 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:37.034 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022827", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:23:33.578629Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:25:37Z" } 2023-09-30T00:25:37.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022827', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.578629Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:25:37.035 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:37.035 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:25:37.035 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:25:37.035 DEBUG:boto:path=/dmuhsu-77/ 2023-09-30T00:25:37.035 DEBUG:boto:auth_path=/dmuhsu-77/ 2023-09-30T00:25:37.036 DEBUG:boto:Method: HEAD 2023-09-30T00:25:37.036 DEBUG:boto:Path: /dmuhsu-77/ 2023-09-30T00:25:37.036 DEBUG:boto:Data: 2023-09-30T00:25:37.036 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.036 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:37.036 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.036 DEBUG:boto:Params: {} 2023-09-30T00:25:37.036 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:25:37.037 DEBUG:boto:Token: None 2023-09-30T00:25:37.037 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-77/ 2023-09-30T00:25:37.037 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dSe7VPlm6GgqzB3+qlnk4ljrwtc= 2023-09-30T00:25:37.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dSe7VPlm6GgqzB3+qlnk4ljrwtc=', 'Content-Length': '0'} 2023-09-30T00:25:37.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0f0516d5f25f8f5-0065176b01-1a63-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.042 DEBUG:boto:path=/dmuhsu-77/obj1 2023-09-30T00:25:37.042 DEBUG:boto:auth_path=/dmuhsu-77/obj1 2023-09-30T00:25:37.042 DEBUG:boto:Method: HEAD 2023-09-30T00:25:37.042 DEBUG:boto:Path: /dmuhsu-77/obj1 2023-09-30T00:25:37.042 DEBUG:boto:Data: 2023-09-30T00:25:37.042 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.043 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:37.043 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.043 DEBUG:boto:Params: {} 2023-09-30T00:25:37.043 DEBUG:boto:Token: None 2023-09-30T00:25:37.043 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-77/obj1 2023-09-30T00:25:37.043 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Hgng5QdkK/qMSNRDD/+5ltzJAn0= 2023-09-30T00:25:37.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Hgng5QdkK/qMSNRDD/+5ltzJAn0=', 'Content-Length': '0'} 2023-09-30T00:25:37.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-77:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.8919.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d8a5101b1708510-0065176b01-1a63-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.045 DEBUG:boto:path=/dmuhsu-77/obj1 2023-09-30T00:25:37.045 DEBUG:boto:auth_path=/dmuhsu-77/obj1 2023-09-30T00:25:37.046 DEBUG:boto:Method: GET 2023-09-30T00:25:37.046 DEBUG:boto:Path: /dmuhsu-77/obj1 2023-09-30T00:25:37.046 DEBUG:boto:Data: 2023-09-30T00:25:37.046 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.046 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:37.046 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.046 DEBUG:boto:Params: {} 2023-09-30T00:25:37.046 DEBUG:boto:Token: None 2023-09-30T00:25:37.047 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-77/obj1 2023-09-30T00:25:37.047 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M0FnOAR1TyK4LHAPV+kOLDmEmT0= 2023-09-30T00:25:37.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M0FnOAR1TyK4LHAPV+kOLDmEmT0=', 'Content-Length': '0'} 2023-09-30T00:25:37.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-77:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.8919.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001944c31c7db8574e-0065176b01-1a63-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.049 DEBUG:boto:path=/dmuhsu-78/ 2023-09-30T00:25:37.049 DEBUG:boto:auth_path=/dmuhsu-78/ 2023-09-30T00:25:37.049 DEBUG:boto:Method: HEAD 2023-09-30T00:25:37.049 DEBUG:boto:Path: /dmuhsu-78/ 2023-09-30T00:25:37.049 DEBUG:boto:Data: 2023-09-30T00:25:37.049 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.049 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:37.049 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.050 DEBUG:boto:Params: {} 2023-09-30T00:25:37.050 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:25:37.050 DEBUG:boto:Token: None 2023-09-30T00:25:37.050 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-78/ 2023-09-30T00:25:37.050 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:eem4upXMIJNN9hDIjyHcRCEGXsM= 2023-09-30T00:25:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eem4upXMIJNN9hDIjyHcRCEGXsM=', 'Content-Length': '0'} 2023-09-30T00:25:37.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ddd5b3047a63bbe-0065176b01-22d4-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.054 DEBUG:boto:path=/dmuhsu-78/obj2 2023-09-30T00:25:37.054 DEBUG:boto:auth_path=/dmuhsu-78/obj2 2023-09-30T00:25:37.055 DEBUG:boto:Method: HEAD 2023-09-30T00:25:37.055 DEBUG:boto:Path: /dmuhsu-78/obj2 2023-09-30T00:25:37.055 DEBUG:boto:Data: 2023-09-30T00:25:37.055 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.055 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:37.055 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.055 DEBUG:boto:Params: {} 2023-09-30T00:25:37.055 DEBUG:boto:Token: None 2023-09-30T00:25:37.055 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-78/obj2 2023-09-30T00:25:37.056 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ojgqax7AJOZJPuf8iH/19yBepZ4= 2023-09-30T00:25:37.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ojgqax7AJOZJPuf8iH/19yBepZ4=', 'Content-Length': '0'} 2023-09-30T00:25:37.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-78:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.8919.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c62928f7d7dfd9ca-0065176b01-22d4-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.057 DEBUG:boto:path=/dmuhsu-78/obj2 2023-09-30T00:25:37.058 DEBUG:boto:auth_path=/dmuhsu-78/obj2 2023-09-30T00:25:37.058 DEBUG:boto:Method: GET 2023-09-30T00:25:37.058 DEBUG:boto:Path: /dmuhsu-78/obj2 2023-09-30T00:25:37.058 DEBUG:boto:Data: 2023-09-30T00:25:37.058 DEBUG:boto:Headers: {} 2023-09-30T00:25:37.058 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:37.058 DEBUG:boto:Port: 8000 2023-09-30T00:25:37.058 DEBUG:boto:Params: {} 2023-09-30T00:25:37.058 DEBUG:boto:Token: None 2023-09-30T00:25:37.059 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:25:37 GMT /dmuhsu-78/obj2 2023-09-30T00:25:37.059 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iTBQkG5EP0DL6U98NdNzbRgaH8w= 2023-09-30T00:25:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iTBQkG5EP0DL6U98NdNzbRgaH8w=', 'Content-Length': '0'} 2023-09-30T00:25:37.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:23:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-78:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.8919.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c02ab165e5218d70-0065176b01-22d4-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:37.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.062 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:37.114 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.114 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.186 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:25:37.187 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-09-30T00:25:37.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.188 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:37.233 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.234 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.274 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.291 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:23:33.638104Z" 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.292 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:37.293 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:25:37.293 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.293 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:25:37.294 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:37.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:25:37.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.295 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:37.345 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.345 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.375 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:37.374+0000 7f9f9ad2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:37.376 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:37.375+0000 7f9f9ad2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:37.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:37.391 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:37.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:37.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:37.396 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:37Z" 2023-09-30T00:25:37.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:37.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:37Z" } 2023-09-30T00:25:37.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:37.399 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:37.399 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:37.399 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:25:37.399 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:25:37.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:37.400 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:37.447 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.447 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:37.499+0000 7fbf2e729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:37.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:25:37.500+0000 7fbf2e729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:37.520 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:25:37.520 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:25:37.520 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.521 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:37.522 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:37.523 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:37.524 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:37.525 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:37.526 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:25:37.527 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:25:37.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:37.528 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:25:37Z" 2023-09-30T00:25:37.528 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:25:37.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:37Z" } 2023-09-30T00:25:37.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:37.529 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:37.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:37.530 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:25:37.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.530 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:37.572 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.572 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.649 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:25:37.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.650 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:37.735 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.735 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.786 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:25:37.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.787 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:37.836 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.836 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:37.911 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:25:37.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:37.911 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:37.960 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:38.025 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:25:38.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.026 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:38.071 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:38.128 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:25:38.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.129 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:38.172 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:38.173 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 12, 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:38.242 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.243 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.244 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:25:38.245 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:25:38.246 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:25:38.248 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.249 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:25:38.250 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow", 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:25:38.251 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:25:38.256 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:25:38.257 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:25:38.258 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:25:38.259 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:25:38.260 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:25:38.261 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:25:38.262 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:25:38.263 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:25:38.264 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:25:38.265 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:38.266 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:38.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:25:38.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:25:38.266 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:25:38.266 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:25:38.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:25:38.267 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:25:38.309 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:38.309 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:38.413 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:25:38.413 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-79 2023-09-30T00:25:38.413 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:25:38.414 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:25:38.414 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.414 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:25:38.414 DEBUG:boto:Data: 2023-09-30T00:25:38.414 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.414 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.414 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.414 DEBUG:boto:Params: {} 2023-09-30T00:25:38.415 DEBUG:boto:Token: None 2023-09-30T00:25:38.415 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/ 2023-09-30T00:25:38.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qG+71h+kdLVvmV1qWt7LbbUn7b8= 2023-09-30T00:25:38.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qG+71h+kdLVvmV1qWt7LbbUn7b8=', 'Content-Length': '0'} 2023-09-30T00:25:38.641 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005de02f0cdae2a7a5-0065176b02-234b-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.641 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:25:38.642 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:25:38.642 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.642 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:25:38.642 DEBUG:boto:Data: 2023-09-30T00:25:38.642 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.642 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.642 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.642 DEBUG:boto:Params: {} 2023-09-30T00:25:38.642 DEBUG:boto:Token: None 2023-09-30T00:25:38.643 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/ 2023-09-30T00:25:38.643 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028= 2023-09-30T00:25:38.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028=', 'Content-Length': '0'} 2023-09-30T00:25:38.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000266ce5a654fefeb1-0065176b02-234b-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.646 DEBUG:boto:path=/dmuhsu-79/obj1 2023-09-30T00:25:38.646 DEBUG:boto:auth_path=/dmuhsu-79/obj1 2023-09-30T00:25:38.646 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.646 DEBUG:boto:Path: /dmuhsu-79/obj1 2023-09-30T00:25:38.647 DEBUG:boto:Data: 2023-09-30T00:25:38.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.647 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.647 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.647 DEBUG:boto:Params: {} 2023-09-30T00:25:38.647 DEBUG:boto:Token: None 2023-09-30T00:25:38.647 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/obj1 2023-09-30T00:25:38.647 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QW8TA9wD19nhiE1CChKrSfZP+ig= 2023-09-30T00:25:38.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QW8TA9wD19nhiE1CChKrSfZP+ig='} 2023-09-30T00:25:38.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d459a929437e8ccd-0065176b02-234b-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.685 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:25:38.685 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:25:38.685 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.685 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:25:38.685 DEBUG:boto:Data: 2023-09-30T00:25:38.685 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.685 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.686 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.686 DEBUG:boto:Params: {} 2023-09-30T00:25:38.686 DEBUG:boto:Token: None 2023-09-30T00:25:38.686 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/ 2023-09-30T00:25:38.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028= 2023-09-30T00:25:38.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028=', 'Content-Length': '0'} 2023-09-30T00:25:38.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc436d656c7b5e23-0065176b02-234b-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.690 DEBUG:boto:path=/dmuhsu-79/obj2 2023-09-30T00:25:38.690 DEBUG:boto:auth_path=/dmuhsu-79/obj2 2023-09-30T00:25:38.690 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.690 DEBUG:boto:Path: /dmuhsu-79/obj2 2023-09-30T00:25:38.691 DEBUG:boto:Data: 2023-09-30T00:25:38.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.691 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.691 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.691 DEBUG:boto:Params: {} 2023-09-30T00:25:38.691 DEBUG:boto:Token: None 2023-09-30T00:25:38.691 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/obj2 2023-09-30T00:25:38.691 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/ulObFv7LYf2oKqNutB9+lZcVEQ= 2023-09-30T00:25:38.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ulObFv7LYf2oKqNutB9+lZcVEQ='} 2023-09-30T00:25:38.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037e0157651c2945d-0065176b02-234b-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.726 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:25:38.726 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:25:38.726 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.726 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:25:38.726 DEBUG:boto:Data: 2023-09-30T00:25:38.726 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.726 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.726 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.727 DEBUG:boto:Params: {} 2023-09-30T00:25:38.727 DEBUG:boto:Token: None 2023-09-30T00:25:38.727 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/ 2023-09-30T00:25:38.727 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028= 2023-09-30T00:25:38.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JR73p8iIBlJXkvB5AQcbwe95028=', 'Content-Length': '0'} 2023-09-30T00:25:38.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2a7d1de44eaee62-0065176b02-234b-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.731 DEBUG:boto:path=/dmuhsu-79/obj3 2023-09-30T00:25:38.731 DEBUG:boto:auth_path=/dmuhsu-79/obj3 2023-09-30T00:25:38.731 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.731 DEBUG:boto:Path: /dmuhsu-79/obj3 2023-09-30T00:25:38.732 DEBUG:boto:Data: 2023-09-30T00:25:38.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.732 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:25:38.732 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.732 DEBUG:boto:Params: {} 2023-09-30T00:25:38.732 DEBUG:boto:Token: None 2023-09-30T00:25:38.732 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-79/obj3 2023-09-30T00:25:38.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0Z5MHgSNUZ4HL0RsYKKqlZ3tDcg= 2023-09-30T00:25:38.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0Z5MHgSNUZ4HL0RsYKKqlZ3tDcg='} 2023-09-30T00:25:38.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9f1590f0059db60-0065176b02-234b-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.779 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-80 2023-09-30T00:25:38.779 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:25:38.779 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:25:38.779 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.779 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:25:38.779 DEBUG:boto:Data: 2023-09-30T00:25:38.780 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.780 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.780 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.780 DEBUG:boto:Params: {} 2023-09-30T00:25:38.780 DEBUG:boto:Token: None 2023-09-30T00:25:38.780 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/ 2023-09-30T00:25:38.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AKpfBzwGLB++B+3EruwHqtthml0= 2023-09-30T00:25:38.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AKpfBzwGLB++B+3EruwHqtthml0=', 'Content-Length': '0'} 2023-09-30T00:25:38.886 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048ff239128f4cc62-0065176b02-1b37-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.886 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:25:38.887 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:25:38.887 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.887 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:25:38.887 DEBUG:boto:Data: 2023-09-30T00:25:38.887 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.887 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.887 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.887 DEBUG:boto:Params: {} 2023-09-30T00:25:38.887 DEBUG:boto:Token: None 2023-09-30T00:25:38.888 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/ 2023-09-30T00:25:38.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw= 2023-09-30T00:25:38.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw=', 'Content-Length': '0'} 2023-09-30T00:25:38.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea94add563f242f8-0065176b02-1b37-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.890 DEBUG:boto:path=/dmuhsu-80/obj4 2023-09-30T00:25:38.891 DEBUG:boto:auth_path=/dmuhsu-80/obj4 2023-09-30T00:25:38.891 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.891 DEBUG:boto:Path: /dmuhsu-80/obj4 2023-09-30T00:25:38.891 DEBUG:boto:Data: 2023-09-30T00:25:38.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.891 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.891 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.891 DEBUG:boto:Params: {} 2023-09-30T00:25:38.891 DEBUG:boto:Token: None 2023-09-30T00:25:38.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/obj4 2023-09-30T00:25:38.892 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:h3klKJJvcIGGIgCl4HhuMu2LheA= 2023-09-30T00:25:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:h3klKJJvcIGGIgCl4HhuMu2LheA='} 2023-09-30T00:25:38.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006442c2ae24ecff61-0065176b02-1b37-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.899 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:25:38.899 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:25:38.899 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.900 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:25:38.900 DEBUG:boto:Data: 2023-09-30T00:25:38.900 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.900 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.900 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.900 DEBUG:boto:Params: {} 2023-09-30T00:25:38.900 DEBUG:boto:Token: None 2023-09-30T00:25:38.900 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/ 2023-09-30T00:25:38.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw= 2023-09-30T00:25:38.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw=', 'Content-Length': '0'} 2023-09-30T00:25:38.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000035395dfbbdaf62a-0065176b02-1b37-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.903 DEBUG:boto:path=/dmuhsu-80/obj5 2023-09-30T00:25:38.903 DEBUG:boto:auth_path=/dmuhsu-80/obj5 2023-09-30T00:25:38.903 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.903 DEBUG:boto:Path: /dmuhsu-80/obj5 2023-09-30T00:25:38.903 DEBUG:boto:Data: 2023-09-30T00:25:38.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.904 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.904 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.904 DEBUG:boto:Params: {} 2023-09-30T00:25:38.904 DEBUG:boto:Token: None 2023-09-30T00:25:38.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/obj5 2023-09-30T00:25:38.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s1UZ8rDsh593XCLgSEZQ4YF+Dl8= 2023-09-30T00:25:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s1UZ8rDsh593XCLgSEZQ4YF+Dl8='} 2023-09-30T00:25:38.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6ab64c642202190-0065176b02-1b37-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.947 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:25:38.947 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:25:38.947 DEBUG:boto:Method: HEAD 2023-09-30T00:25:38.947 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:25:38.947 DEBUG:boto:Data: 2023-09-30T00:25:38.947 DEBUG:boto:Headers: {} 2023-09-30T00:25:38.948 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.948 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.948 DEBUG:boto:Params: {} 2023-09-30T00:25:38.948 DEBUG:boto:Token: None 2023-09-30T00:25:38.948 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/ 2023-09-30T00:25:38.948 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw= 2023-09-30T00:25:38.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f5ekRsWSWlMTE+tHEOdifkjfOEw=', 'Content-Length': '0'} 2023-09-30T00:25:38.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000865c6a4e211ef9e8-0065176b02-1b37-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.950 DEBUG:boto:path=/dmuhsu-80/obj6 2023-09-30T00:25:38.951 DEBUG:boto:auth_path=/dmuhsu-80/obj6 2023-09-30T00:25:38.951 DEBUG:boto:Method: PUT 2023-09-30T00:25:38.951 DEBUG:boto:Path: /dmuhsu-80/obj6 2023-09-30T00:25:38.951 DEBUG:boto:Data: 2023-09-30T00:25:38.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:25:38.951 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:25:38.951 DEBUG:boto:Port: 8000 2023-09-30T00:25:38.951 DEBUG:boto:Params: {} 2023-09-30T00:25:38.951 DEBUG:boto:Token: None 2023-09-30T00:25:38.952 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:25:38 GMT /dmuhsu-80/obj6 2023-09-30T00:25:38.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QroudUYhtGwErfFGajYu47z7Vxs= 2023-09-30T00:25:38.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:25:38 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QroudUYhtGwErfFGajYu47z7Vxs='} 2023-09-30T00:25:38.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c24691bc450149c-0065176b02-1b37-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:25:38.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:38.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:38.959 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:39.022 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:39.023 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:39.116 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:39.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:25:38.862705Z" 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:25:39.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:25:39.156 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:25:39.156 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:39.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:25:39.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:39.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:39.157 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:39.203 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:39.203 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:39.234 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:39.233+0000 7fc39374b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:39.235 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:39.234+0000 7fc39374b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:39.248 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:39.248 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:39.249 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:39Z" 2023-09-30T00:25:39.255 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:39.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:39Z" } 2023-09-30T00:25:39.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:39.256 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:39.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:39.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:25:44.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:44.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:44.258 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:44.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:44.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:44.344 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:44.343+0000 7efd16fa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:44.345 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:44.344+0000 7efd16fa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:44.359 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:44.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:44.360 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:44.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:44.362 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:44.363 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:44.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.365 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:44.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:44.366 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:44.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:44.366 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:44Z" 2023-09-30T00:25:44.366 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:44.367 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:44Z" } 2023-09-30T00:25:44.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:44.368 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:44.368 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:44.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:25:49.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:49.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:49.370 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:49.425 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:49.425 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:49.458 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:49.457+0000 7fde9fb20880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:49.459 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:49.458+0000 7fde9fb20880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:49.473 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:49.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:49.475 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:49.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:49.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:49Z" 2023-09-30T00:25:49.480 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:49.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:49Z" } 2023-09-30T00:25:49.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:49.482 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:49.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:49.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:25:54.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:54.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:54.483 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:54.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:54.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:54.570 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:54.569+0000 7f6b6a51e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:54.570 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:54.570+0000 7f6b6a51e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:54.585 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:54.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:54.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:54.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:54.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:54.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:54.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.595 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:54.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:54.595 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:54.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:54.595 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:54Z" 2023-09-30T00:25:54.596 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:54.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:54Z" } 2023-09-30T00:25:54.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:54.597 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:54.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:54.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:25:59.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:59.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:25:59.599 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:25:59.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:25:59.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:25:59.686 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:59.685+0000 7f44da411880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:59.687 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:25:59.686+0000 7f44da411880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:25:59.701 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:25:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:25:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:25:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:25:59Z" 2023-09-30T00:25:59.708 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:25:59.709 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:25:59Z" } 2023-09-30T00:25:59.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:25:59.710 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:59.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:25:59.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:04.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:04.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:04.712 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:04.799 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:04.798+0000 7fde4b74b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:04.799 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:04.799+0000 7fde4b74b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:04.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:04.818 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:04.818 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:04.819 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:04.819 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:04.819 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:04.819 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:04.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.820 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:04.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:04.820 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:04.820 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:04.820 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:04.821 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:04.821 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:04.821 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:04.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:04.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:04.822 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:04.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:04.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:04.823 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:04.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:04.825 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:04Z" 2023-09-30T00:26:04.827 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:04.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:04Z" } 2023-09-30T00:26:04.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:04.829 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:04.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:04.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:09.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:09.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:09.831 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:09.884 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:09.884 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:09.917 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:09.917+0000 7fd15d88b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:09.918 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:09.917+0000 7fd15d88b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:09.933 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:09.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:09.935 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:09.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:09Z" 2023-09-30T00:26:09.939 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:09.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:09Z" } 2023-09-30T00:26:09.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:09.941 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:09.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:09.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:14.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:14.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:14.942 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:14.996 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:14.996 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:15.029 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:15.028+0000 7fa27ce0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:15.030 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:15.029+0000 7fa27ce0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:15.044 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:15.045 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:15.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:15.047 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:15.048 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:15.049 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:15Z" 2023-09-30T00:26:15.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:15.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:15Z" } 2023-09-30T00:26:15.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:15.052 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:15.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:15.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:20.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:20.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:20.054 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:20.107 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:20.107 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:20.140 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:20.139+0000 7ff80bf75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:20.141 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:20.140+0000 7ff80bf75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:20.155 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:20.155 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:20.155 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:20.156 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:20.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:20.158 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:20.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:20.160 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:20.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:20Z" 2023-09-30T00:26:20.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:20.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:20Z" } 2023-09-30T00:26:20.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:20.164 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:20.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:20.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:25.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:25.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:25.166 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:25.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:25.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:25.255 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:25.254+0000 7f0b3ea19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:25.255 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:25.255+0000 7f0b3ea19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:25.270 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:25.270 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:25.270 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:25.270 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:25.271 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:25.272 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:25.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:25.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:25.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:25.276 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:25Z" 2023-09-30T00:26:25.277 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:25.277 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:25Z" } 2023-09-30T00:26:25.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:25.278 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:25.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:25.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:30.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:30.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:30.280 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:30.332 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:30.332 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:30.365 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:30.365+0000 7fe39d611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:30.366 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:30.365+0000 7fe39d611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:30.381 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:30.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:30.383 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:30.384 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:30.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:30Z" 2023-09-30T00:26:30.387 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:30.388 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:30Z" } 2023-09-30T00:26:30.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:30.389 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:30.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:30.389 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:35.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:35.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:35.391 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:35.444 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:35.444 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:35.477 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:35.476+0000 7f7a8b926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:35.478 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:35.477+0000 7f7a8b926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:35.493 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:35.493 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:35.493 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:35.493 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:35.493 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:35.494 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:35.494 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:35.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.495 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:35.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:35.495 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:35.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:35.497 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:35.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:35.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:35.502 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:35.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:35.503 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:35Z" 2023-09-30T00:26:35.503 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:35.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:35Z" } 2023-09-30T00:26:35.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:35.505 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:35.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:35.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:40.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:40.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:40.506 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:40.560 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:40.560 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:40.594 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:40.593+0000 7f2fcf722880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:40.595 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:40.594+0000 7f2fcf722880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:40.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:40.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:40.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:40.613 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:40.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:40.615 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:40Z" 2023-09-30T00:26:40.616 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:40.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:40Z" } 2023-09-30T00:26:40.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:40.617 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:40.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:40.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:45.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:45.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:45.619 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:45.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:45.672 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:45.705 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:45.705+0000 7fd3fd410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:45.706 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:45.705+0000 7fd3fd410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:45.721 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:45.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:45.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:45.724 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:45.725 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:45.726 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:45Z" 2023-09-30T00:26:45.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:45.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:45Z" } 2023-09-30T00:26:45.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:45.728 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:45.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:45.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:50.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:50.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:50.730 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:50.782 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:50.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:50.816 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:50.815+0000 7f07f6819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:50.817 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:50.816+0000 7f07f6819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:50.831 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:50.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:50.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:50.834 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:50Z" 2023-09-30T00:26:50.838 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:50.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:50Z" } 2023-09-30T00:26:50.839 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:50.839 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:50.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:50.839 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:26:55.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:55.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:26:55.841 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:26:55.894 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:26:55.895 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:26:55.928 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:55.928+0000 7f245360f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:55.929 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:26:55.929+0000 7f245360f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:26:55.943 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:26:55.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:55.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:26:55.946 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:55.947 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:26:55.948 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:26:55.949 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:26:55Z" 2023-09-30T00:26:55.950 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:26:55.951 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:26:55Z" } 2023-09-30T00:26:55.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:26:55.951 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:55.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:26:55.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:00.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:00.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:00.953 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:01.006 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:01.006 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:01.039 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:01.039+0000 7ff661f24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:01.040 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:01.040+0000 7ff661f24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:01.055 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:01.055 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:01.055 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:01.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:01.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:01.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:01.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:01.059 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:01.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:27:01.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:01.062 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:01Z" 2023-09-30T00:27:01.062 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:01.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:01Z" } 2023-09-30T00:27:01.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:01.063 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:01.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:01.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:06.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:06.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:06.065 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:06.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:06.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:06.152 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:06.151+0000 7ff1d9721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:06.152 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:06.152+0000 7ff1d9721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:06.167 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:06.167 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:06.167 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:06.167 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:06.168 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:06.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:06.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:06.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:06.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:06Z" 2023-09-30T00:27:06.173 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:06.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:06Z" } 2023-09-30T00:27:06.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:06.175 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:06.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:06.175 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:11.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:11.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:11.177 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:11.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:11.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:11.265 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:11.264+0000 7f7e2d511880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:11.265 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:11.265+0000 7f7e2d511880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:11.280 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:11.280 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:11.280 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:11.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:11.281 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:11.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:11.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:11.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:11Z" 2023-09-30T00:27:11.286 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:11.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:11Z" } 2023-09-30T00:27:11.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:11.288 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:11.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:11.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:16.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:16.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:16.290 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:16.343 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:16.343 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:16.377 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:16.377+0000 7fb1a4032880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:16.378 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:16.378+0000 7fb1a4032880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:16.393 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:16.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:16.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:16.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:16.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:16.398 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:16Z" 2023-09-30T00:27:16.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:16.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:16Z" } 2023-09-30T00:27:16.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:16.400 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:16.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:16.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:21.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:21.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:21.402 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:21.457 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:21.457 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:21.491 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:21.490+0000 7f6381467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:21.492 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:27:21.491+0000 7f6381467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:21.506 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.507 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:21.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:21.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:21.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:21.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:21.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:21.513 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:27:21.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:21Z" 2023-09-30T00:27:21.515 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:21.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:21Z" } 2023-09-30T00:27:21.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:21.517 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:21.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:21.517 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:27:21.517 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:27:21.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:21.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:21.518 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:21.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:21.570 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:21.606 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:21.605+0000 7f5d41a0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:21.607 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:21.606+0000 7f5d41a0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:21.623 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:21.623 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:21.623 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:21.624 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:21.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:21.626 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:21.627 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:21.628 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:21.629 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:21.630 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:21.630 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:21.631 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:27:21.632 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:21.633 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:21Z" 2023-09-30T00:27:21.633 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:21.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:21Z" } 2023-09-30T00:27:21.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:21.634 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:21.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:21.635 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:26.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:26.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:26.636 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:26.688 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:26.688 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:26.746 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:26.745+0000 7f861d536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:26.747 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:26.746+0000 7f861d536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:26.774 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:27:26.775 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:26.776 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:26.777 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033413.638104_826.1", 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:23:33.638104Z", 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:26.778 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:26.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:26Z" 2023-09-30T00:27:26.780 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:26.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033413.638104_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:23:33.638104Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:26Z" } 2023-09-30T00:27:26.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033413.638104_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.638104Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:26.781 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:26.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033413.638104_826.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:26.782 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033538.862705_872.1 target=1_1696033413.638104_826.1 2023-09-30T00:27:31.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:31.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:31.783 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:31.835 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:31.836 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:31.887 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:31.886+0000 7f1efcc67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:31.888 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:27:31.887+0000 7f1efcc67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:27:31.904 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:31.904 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:31.905 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:31.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:27:31.907 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:31.908 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:31.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:27:31.910 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:31.911 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:31Z" 2023-09-30T00:27:31.911 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:31.911 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:27:31Z" } 2023-09-30T00:27:31.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:27:31.912 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:31.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:27:31.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:27:31.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:31.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:31.913 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:31.959 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:31.959 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:32.018 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:27:32.018 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:32.018 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:27:32.018 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000023107", 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:25:38.751404Z" 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:32.019 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:32.020 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000017914", 2023-09-30T00:27:32.020 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:25:38.717908Z" 2023-09-30T00:27:32.020 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:32.020 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:27:32.021 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000023107', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017914'} 2023-09-30T00:27:32.021 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:27:32.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:32.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:32.021 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:32.075 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:32.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:32.126 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022827", 2023-09-30T00:27:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.578629Z" 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:27:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:27:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:32Z" 2023-09-30T00:27:32.132 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:32.133 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022827", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:23:33.578629Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:27:32Z" } 2023-09-30T00:27:32.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022827', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.578629Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:27:32.133 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000023107', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017914'} 2023-09-30T00:27:32.133 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:27:32.133 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023107 target=00000000000000000000:00000000000000022827, shard=3 master=00000000000000000000:00000000000000017914 target=00000000000000000000:00000000000000017632 2023-09-30T00:27:37.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:37.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:37.135 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:37.190 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:37.191 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:37.242 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:37.242 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:37.242 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:37.242 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:27:37.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:37.244 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000022827", 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:23:33.578629Z" 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:37.245 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:37.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017632", 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:37.247 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.596706Z" 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:37Z" 2023-09-30T00:27:37.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:37.249 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022827", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:23:33.578629Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.596706Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:27:37Z" } 2023-09-30T00:27:37.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022827', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:23:33.578629Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.596706Z'}}] 2023-09-30T00:27:37.250 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000023107', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017914'} 2023-09-30T00:27:37.250 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000022827', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017632'} 2023-09-30T00:27:37.250 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023107 target=00000000000000000000:00000000000000022827, shard=3 master=00000000000000000000:00000000000000017914 target=00000000000000000000:00000000000000017632 2023-09-30T00:27:42.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.252 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:42.308 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:42.309 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:42.359 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:27:42.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:27:42.359 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:27:42.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:42.360 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013089", 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:11:00.588242Z" 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.361 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023107", 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.751404Z" 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:42.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000017914", 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:27:42.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.717908Z" 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.365 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:27:42Z" 2023-09-30T00:27:42.366 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:27:42.367 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013089", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:11:00.588242Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023107", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:25:38.751404Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.717908Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:27:42Z" } 2023-09-30T00:27:42.367 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013089', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:11:00.588242Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023107', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.751404Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.717908Z'}}] 2023-09-30T00:27:42.367 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000023107', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017914'} 2023-09-30T00:27:42.367 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013089', 1: '00000000000000000000:00000000000000023107', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000017914'} 2023-09-30T00:27:42.367 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:27:42.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.368 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:42.413 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:42.414 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:42.463 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2023-09-30T00:27:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000010370", 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-30T00:27:39.742807Z" 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:27:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006160", 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2023-09-30T00:27:39.741426Z" 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:27:42.465 INFO:teuthology.orchestra.run.smithi184.stdout:] 2023-09-30T00:27:42.466 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:27:42.466 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2023-09-30T00:27:42.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:42.467 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:42.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:42.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-30T00:27:42.611 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:42.612 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010090", 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:42.613 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.910650Z" 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:42.614 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006020", 2023-09-30T00:27:42.615 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.957983Z" 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-30T00:27:42.616 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:42.617 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:42Z" 2023-09-30T00:27:42.617 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:42.618 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.899074Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010090", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.910650Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:37.239753Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.957983Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-30T00:27:42Z" } 2023-09-30T00:27:42.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.899074Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010090', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.910650Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:37.239753Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.957983Z'}}] 2023-09-30T00:27:42.618 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:27:42.618 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010090', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006020'} 2023-09-30T00:27:42.618 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000010370 target=00000000000000000000:00000000000000010090, shard=3 master=00000000000000000000:00000000000000006160 target=00000000000000000000:00000000000000006020 2023-09-30T00:27:47.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:47.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:47.620 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:47.675 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:47.675 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:47.737 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:47.738 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:47.739 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010090", 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.910650Z" 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:47.740 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:47.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006020", 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.957983Z" 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:47.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:47Z" 2023-09-30T00:27:47.743 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:47.744 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.899074Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010090", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.910650Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:37.239753Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.957983Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-30T00:27:47Z" } 2023-09-30T00:27:47.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.899074Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010090', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.910650Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:37.239753Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.957983Z'}}] 2023-09-30T00:27:47.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:27:47.744 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010090', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006020'} 2023-09-30T00:27:47.745 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000010370 target=00000000000000000000:00000000000000010090, shard=3 master=00000000000000000000:00000000000000006160 target=00000000000000000000:00000000000000006020 2023-09-30T00:27:52.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:52.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:52.746 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:52.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:52.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:52.861 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:52.862 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010090", 2023-09-30T00:27:52.863 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.910650Z" 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:52.864 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:52.865 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006020", 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.957983Z" 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:52.866 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-30T00:27:52.867 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-30T00:27:52.867 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:52.867 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:52Z" 2023-09-30T00:27:52.867 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:52.868 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.899074Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010090", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.910650Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:37.239753Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.957983Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-30T00:27:52Z" } 2023-09-30T00:27:52.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.899074Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010090', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.910650Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:37.239753Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.957983Z'}}] 2023-09-30T00:27:52.868 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:27:52.868 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010090', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006020'} 2023-09-30T00:27:52.868 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000010370 target=00000000000000000000:00000000000000010090, shard=3 master=00000000000000000000:00000000000000006160 target=00000000000000000000:00000000000000006020 2023-09-30T00:27:57.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:57.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:27:57.870 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:27:57.927 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:27:57.928 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:27:58.010 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:27:58.010 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:27:58.010 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:27:58.010 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:27:58.010 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:58.011 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010090", 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:58.012 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.910650Z" 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:27:58.013 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:27:58.014 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006020", 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.957983Z" 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:27:58.015 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-30T00:27:58.016 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-30T00:27:58.016 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:27:58.016 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:27:57Z" 2023-09-30T00:27:58.016 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:27:58.017 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.899074Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010090", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.910650Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:37.239753Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.957983Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-30T00:27:57Z" } 2023-09-30T00:27:58.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.899074Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010090', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.910650Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:37.239753Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.957983Z'}}] 2023-09-30T00:27:58.017 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:27:58.017 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010090', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006020'} 2023-09-30T00:27:58.017 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000010370 target=00000000000000000000:00000000000000010090, shard=3 master=00000000000000000000:00000000000000006160 target=00000000000000000000:00000000000000006020 2023-09-30T00:28:03.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.019 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.073 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.133 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:28:03.133 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:28:03.133 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:28:03.133 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:28:03.133 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 14475923928077737876 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000005441", 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.134 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.899074Z" 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010370", 2023-09-30T00:28:03.135 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:27:39.742807Z" 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000010654", 2023-09-30T00:28:03.136 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:21:37.239753Z" 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2023-09-30T00:28:03.137 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000006160", 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:27:39.741426Z" 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2023-09-30T00:28:03.138 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2023-09-30T00:28:03.139 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.139 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:28:03Z" 2023-09-30T00:28:03.139 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:28:03.140 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14475923928077737876 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.899074Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010370", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.742807Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:21:37.239753Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006160", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.741426Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-30T00:28:03Z" } 2023-09-30T00:28:03.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.899074Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010370', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.742807Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:21:37.239753Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006160', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.741426Z'}}] 2023-09-30T00:28:03.140 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:28:03.140 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000010370', 2: '00000000000000000000:00000000000000010654', 3: '00000000000000000000:00000000000000006160'} 2023-09-30T00:28:03.140 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2023-09-30T00:28:03.141 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:28:03.141 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:28:03.141 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.141 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:28:03.141 DEBUG:boto:Data: 2023-09-30T00:28:03.141 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.141 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.141 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.141 DEBUG:boto:Params: {} 2023-09-30T00:28:03.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:28:03.142 DEBUG:boto:Token: None 2023-09-30T00:28:03.142 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/ 2023-09-30T00:28:03.142 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fOzhbryE4uGPdUWWg1ZraX8jg3k= 2023-09-30T00:28:03.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fOzhbryE4uGPdUWWg1ZraX8jg3k=', 'Content-Length': '0'} 2023-09-30T00:28:03.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db30fbb0ecf7c57c-0065176b93-1b37-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.147 DEBUG:boto:path=/dmuhsu-79/obj1 2023-09-30T00:28:03.147 DEBUG:boto:auth_path=/dmuhsu-79/obj1 2023-09-30T00:28:03.147 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.147 DEBUG:boto:Path: /dmuhsu-79/obj1 2023-09-30T00:28:03.147 DEBUG:boto:Data: 2023-09-30T00:28:03.147 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.147 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.147 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.147 DEBUG:boto:Params: {} 2023-09-30T00:28:03.148 DEBUG:boto:Token: None 2023-09-30T00:28:03.148 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj1 2023-09-30T00:28:03.148 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:74gMQmNBmH2YQlG8loli34WaWc4= 2023-09-30T00:28:03.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:74gMQmNBmH2YQlG8loli34WaWc4=', 'Content-Length': '0'} 2023-09-30T00:28:03.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cffc76464f83a9b5-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.150 DEBUG:boto:path=/dmuhsu-79/obj1 2023-09-30T00:28:03.150 DEBUG:boto:auth_path=/dmuhsu-79/obj1 2023-09-30T00:28:03.150 DEBUG:boto:Method: GET 2023-09-30T00:28:03.150 DEBUG:boto:Path: /dmuhsu-79/obj1 2023-09-30T00:28:03.150 DEBUG:boto:Data: 2023-09-30T00:28:03.151 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.151 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.151 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.151 DEBUG:boto:Params: {} 2023-09-30T00:28:03.151 DEBUG:boto:Token: None 2023-09-30T00:28:03.151 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj1 2023-09-30T00:28:03.151 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rIFH8lJSziar+ZmyQzu5RMeZbj8= 2023-09-30T00:28:03.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rIFH8lJSziar+ZmyQzu5RMeZbj8=', 'Content-Length': '0'} 2023-09-30T00:28:03.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d3372d4a4a44b4d-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.153 DEBUG:boto:path=/dmuhsu-79/obj2 2023-09-30T00:28:03.153 DEBUG:boto:auth_path=/dmuhsu-79/obj2 2023-09-30T00:28:03.153 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.154 DEBUG:boto:Path: /dmuhsu-79/obj2 2023-09-30T00:28:03.154 DEBUG:boto:Data: 2023-09-30T00:28:03.154 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.154 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.154 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.154 DEBUG:boto:Params: {} 2023-09-30T00:28:03.154 DEBUG:boto:Token: None 2023-09-30T00:28:03.154 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj2 2023-09-30T00:28:03.154 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zRnUZ7+jHNiOst9w5PzFhCblUo0= 2023-09-30T00:28:03.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zRnUZ7+jHNiOst9w5PzFhCblUo0=', 'Content-Length': '0'} 2023-09-30T00:28:03.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013bf2a1d07cf5949-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.158 DEBUG:boto:path=/dmuhsu-79/obj2 2023-09-30T00:28:03.158 DEBUG:boto:auth_path=/dmuhsu-79/obj2 2023-09-30T00:28:03.158 DEBUG:boto:Method: GET 2023-09-30T00:28:03.158 DEBUG:boto:Path: /dmuhsu-79/obj2 2023-09-30T00:28:03.158 DEBUG:boto:Data: 2023-09-30T00:28:03.158 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.158 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.159 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.159 DEBUG:boto:Params: {} 2023-09-30T00:28:03.159 DEBUG:boto:Token: None 2023-09-30T00:28:03.159 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj2 2023-09-30T00:28:03.159 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bzbVum3V2sOkfPYiG3Yoirpmqbw= 2023-09-30T00:28:03.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bzbVum3V2sOkfPYiG3Yoirpmqbw=', 'Content-Length': '0'} 2023-09-30T00:28:03.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087a7ea9e239b025c-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.161 DEBUG:boto:path=/dmuhsu-79/obj3 2023-09-30T00:28:03.161 DEBUG:boto:auth_path=/dmuhsu-79/obj3 2023-09-30T00:28:03.161 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.161 DEBUG:boto:Path: /dmuhsu-79/obj3 2023-09-30T00:28:03.161 DEBUG:boto:Data: 2023-09-30T00:28:03.161 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.162 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.162 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.162 DEBUG:boto:Params: {} 2023-09-30T00:28:03.162 DEBUG:boto:Token: None 2023-09-30T00:28:03.162 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj3 2023-09-30T00:28:03.162 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Gi3j+jpUUqr3MaiAnqmSFBbWvZI= 2023-09-30T00:28:03.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Gi3j+jpUUqr3MaiAnqmSFBbWvZI=', 'Content-Length': '0'} 2023-09-30T00:28:03.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c251bb117e5d5f5e-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.164 DEBUG:boto:path=/dmuhsu-79/obj3 2023-09-30T00:28:03.165 DEBUG:boto:auth_path=/dmuhsu-79/obj3 2023-09-30T00:28:03.165 DEBUG:boto:Method: GET 2023-09-30T00:28:03.165 DEBUG:boto:Path: /dmuhsu-79/obj3 2023-09-30T00:28:03.165 DEBUG:boto:Data: 2023-09-30T00:28:03.165 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.165 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:03.165 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.165 DEBUG:boto:Params: {} 2023-09-30T00:28:03.165 DEBUG:boto:Token: None 2023-09-30T00:28:03.165 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj3 2023-09-30T00:28:03.166 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Z4zP8WkEkV6pIxPlwB5+6hcldrk= 2023-09-30T00:28:03.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z4zP8WkEkV6pIxPlwB5+6hcldrk=', 'Content-Length': '0'} 2023-09-30T00:28:03.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-79:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015263fe52fc1402f-0065176b93-1b37-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.168 DEBUG:boto:path=/dmuhsu-79/ 2023-09-30T00:28:03.168 DEBUG:boto:auth_path=/dmuhsu-79/ 2023-09-30T00:28:03.168 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.168 DEBUG:boto:Path: /dmuhsu-79/ 2023-09-30T00:28:03.168 DEBUG:boto:Data: 2023-09-30T00:28:03.168 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.168 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.169 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.169 DEBUG:boto:Params: {} 2023-09-30T00:28:03.169 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:28:03.169 DEBUG:boto:Token: None 2023-09-30T00:28:03.169 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/ 2023-09-30T00:28:03.169 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fOzhbryE4uGPdUWWg1ZraX8jg3k= 2023-09-30T00:28:03.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fOzhbryE4uGPdUWWg1ZraX8jg3k=', 'Content-Length': '0'} 2023-09-30T00:28:03.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f281d54409ad3c8a-0065176b93-234d-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.174 DEBUG:boto:path=/dmuhsu-79/obj1 2023-09-30T00:28:03.174 DEBUG:boto:auth_path=/dmuhsu-79/obj1 2023-09-30T00:28:03.174 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.174 DEBUG:boto:Path: /dmuhsu-79/obj1 2023-09-30T00:28:03.174 DEBUG:boto:Data: 2023-09-30T00:28:03.174 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.175 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.175 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.175 DEBUG:boto:Params: {} 2023-09-30T00:28:03.175 DEBUG:boto:Token: None 2023-09-30T00:28:03.175 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj1 2023-09-30T00:28:03.175 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:74gMQmNBmH2YQlG8loli34WaWc4= 2023-09-30T00:28:03.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:74gMQmNBmH2YQlG8loli34WaWc4=', 'Content-Length': '0'} 2023-09-30T00:28:03.176 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005ff5cf5706291ed2-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.177 DEBUG:boto:path=/dmuhsu-79/obj2 2023-09-30T00:28:03.177 DEBUG:boto:auth_path=/dmuhsu-79/obj2 2023-09-30T00:28:03.177 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.177 DEBUG:boto:Path: /dmuhsu-79/obj2 2023-09-30T00:28:03.177 DEBUG:boto:Data: 2023-09-30T00:28:03.177 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.177 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.177 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.177 DEBUG:boto:Params: {} 2023-09-30T00:28:03.178 DEBUG:boto:Token: None 2023-09-30T00:28:03.178 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj2 2023-09-30T00:28:03.178 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zRnUZ7+jHNiOst9w5PzFhCblUo0= 2023-09-30T00:28:03.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zRnUZ7+jHNiOst9w5PzFhCblUo0=', 'Content-Length': '0'} 2023-09-30T00:28:03.179 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000064236ccc12eb85ed-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.180 DEBUG:boto:path=/dmuhsu-79/obj3 2023-09-30T00:28:03.180 DEBUG:boto:auth_path=/dmuhsu-79/obj3 2023-09-30T00:28:03.180 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.180 DEBUG:boto:Path: /dmuhsu-79/obj3 2023-09-30T00:28:03.180 DEBUG:boto:Data: 2023-09-30T00:28:03.180 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.180 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.180 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.180 DEBUG:boto:Params: {} 2023-09-30T00:28:03.180 DEBUG:boto:Token: None 2023-09-30T00:28:03.181 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-79/obj3 2023-09-30T00:28:03.181 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Gi3j+jpUUqr3MaiAnqmSFBbWvZI= 2023-09-30T00:28:03.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Gi3j+jpUUqr3MaiAnqmSFBbWvZI=', 'Content-Length': '0'} 2023-09-30T00:28:03.182 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a8a9b5d3991c83d4-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.182 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:28:03.182 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:28:03.182 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.183 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:28:03.183 DEBUG:boto:Data: 2023-09-30T00:28:03.183 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.183 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.183 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.183 DEBUG:boto:Params: {} 2023-09-30T00:28:03.183 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:28:03.183 DEBUG:boto:Token: None 2023-09-30T00:28:03.184 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/ 2023-09-30T00:28:03.184 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:67cQgPwqrE1dxLOslxtFfJSqDbE= 2023-09-30T00:28:03.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:67cQgPwqrE1dxLOslxtFfJSqDbE=', 'Content-Length': '0'} 2023-09-30T00:28:03.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a0e7a18f0aa729a-0065176b93-234b-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.189 DEBUG:boto:path=/dmuhsu-80/obj4 2023-09-30T00:28:03.189 DEBUG:boto:auth_path=/dmuhsu-80/obj4 2023-09-30T00:28:03.190 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.190 DEBUG:boto:Path: /dmuhsu-80/obj4 2023-09-30T00:28:03.190 DEBUG:boto:Data: 2023-09-30T00:28:03.190 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.190 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.190 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.190 DEBUG:boto:Params: {} 2023-09-30T00:28:03.190 DEBUG:boto:Token: None 2023-09-30T00:28:03.190 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj4 2023-09-30T00:28:03.191 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJOBCexsJWdB/8hi7YULmWK+viA= 2023-09-30T00:28:03.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJOBCexsJWdB/8hi7YULmWK+viA=', 'Content-Length': '0'} 2023-09-30T00:28:03.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aa30f1643db4d63-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.193 DEBUG:boto:path=/dmuhsu-80/obj4 2023-09-30T00:28:03.193 DEBUG:boto:auth_path=/dmuhsu-80/obj4 2023-09-30T00:28:03.193 DEBUG:boto:Method: GET 2023-09-30T00:28:03.193 DEBUG:boto:Path: /dmuhsu-80/obj4 2023-09-30T00:28:03.193 DEBUG:boto:Data: 2023-09-30T00:28:03.193 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.193 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.194 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.194 DEBUG:boto:Params: {} 2023-09-30T00:28:03.194 DEBUG:boto:Token: None 2023-09-30T00:28:03.194 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj4 2023-09-30T00:28:03.194 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2QNckT6r62dvH1IOtv4oEhAPQa0= 2023-09-30T00:28:03.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2QNckT6r62dvH1IOtv4oEhAPQa0=', 'Content-Length': '0'} 2023-09-30T00:28:03.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f2eb9156c39ef89-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.196 DEBUG:boto:path=/dmuhsu-80/obj5 2023-09-30T00:28:03.196 DEBUG:boto:auth_path=/dmuhsu-80/obj5 2023-09-30T00:28:03.196 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.197 DEBUG:boto:Path: /dmuhsu-80/obj5 2023-09-30T00:28:03.197 DEBUG:boto:Data: 2023-09-30T00:28:03.197 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.197 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.197 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.197 DEBUG:boto:Params: {} 2023-09-30T00:28:03.197 DEBUG:boto:Token: None 2023-09-30T00:28:03.197 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj5 2023-09-30T00:28:03.197 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QjtKIIayc312CdKUaWrmmjgroC0= 2023-09-30T00:28:03.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QjtKIIayc312CdKUaWrmmjgroC0=', 'Content-Length': '0'} 2023-09-30T00:28:03.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001eb9aa5c63d8f567-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.199 DEBUG:boto:path=/dmuhsu-80/obj5 2023-09-30T00:28:03.200 DEBUG:boto:auth_path=/dmuhsu-80/obj5 2023-09-30T00:28:03.200 DEBUG:boto:Method: GET 2023-09-30T00:28:03.200 DEBUG:boto:Path: /dmuhsu-80/obj5 2023-09-30T00:28:03.200 DEBUG:boto:Data: 2023-09-30T00:28:03.200 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.200 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.200 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.200 DEBUG:boto:Params: {} 2023-09-30T00:28:03.200 DEBUG:boto:Token: None 2023-09-30T00:28:03.201 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj5 2023-09-30T00:28:03.201 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:R0utRkVz6KNj1gsxAwvhZYaSHow= 2023-09-30T00:28:03.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R0utRkVz6KNj1gsxAwvhZYaSHow=', 'Content-Length': '0'} 2023-09-30T00:28:03.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000680d9aca0a25cf5d-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.202 DEBUG:boto:path=/dmuhsu-80/obj6 2023-09-30T00:28:03.203 DEBUG:boto:auth_path=/dmuhsu-80/obj6 2023-09-30T00:28:03.203 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.203 DEBUG:boto:Path: /dmuhsu-80/obj6 2023-09-30T00:28:03.203 DEBUG:boto:Data: 2023-09-30T00:28:03.203 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.203 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.203 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.203 DEBUG:boto:Params: {} 2023-09-30T00:28:03.204 DEBUG:boto:Token: None 2023-09-30T00:28:03.204 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj6 2023-09-30T00:28:03.204 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T8TKUdJuLNKhm2Em6cGj/kbQV5I= 2023-09-30T00:28:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T8TKUdJuLNKhm2Em6cGj/kbQV5I=', 'Content-Length': '0'} 2023-09-30T00:28:03.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000512494c82cb6b3a3-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.206 DEBUG:boto:path=/dmuhsu-80/obj6 2023-09-30T00:28:03.206 DEBUG:boto:auth_path=/dmuhsu-80/obj6 2023-09-30T00:28:03.206 DEBUG:boto:Method: GET 2023-09-30T00:28:03.206 DEBUG:boto:Path: /dmuhsu-80/obj6 2023-09-30T00:28:03.206 DEBUG:boto:Data: 2023-09-30T00:28:03.206 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.206 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:03.206 DEBUG:boto:Port: 8000 2023-09-30T00:28:03.206 DEBUG:boto:Params: {} 2023-09-30T00:28:03.207 DEBUG:boto:Token: None 2023-09-30T00:28:03.207 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj6 2023-09-30T00:28:03.207 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xhvFsW3ga9G0LWEi1SHQqiwtXz8= 2023-09-30T00:28:03.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xhvFsW3ga9G0LWEi1SHQqiwtXz8=', 'Content-Length': '0'} 2023-09-30T00:28:03.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-80:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9039.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046a31caa62d29630-0065176b93-234b-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.250 DEBUG:boto:path=/dmuhsu-80/ 2023-09-30T00:28:03.251 DEBUG:boto:auth_path=/dmuhsu-80/ 2023-09-30T00:28:03.251 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.251 DEBUG:boto:Path: /dmuhsu-80/ 2023-09-30T00:28:03.251 DEBUG:boto:Data: 2023-09-30T00:28:03.251 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.251 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.251 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.252 DEBUG:boto:Params: {} 2023-09-30T00:28:03.252 DEBUG:boto:Token: None 2023-09-30T00:28:03.252 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/ 2023-09-30T00:28:03.252 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:67cQgPwqrE1dxLOslxtFfJSqDbE= 2023-09-30T00:28:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:67cQgPwqrE1dxLOslxtFfJSqDbE=', 'Content-Length': '0'} 2023-09-30T00:28:03.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b32ac6c42e16859f-0065176b93-234d-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.254 DEBUG:boto:path=/dmuhsu-80/obj4 2023-09-30T00:28:03.255 DEBUG:boto:auth_path=/dmuhsu-80/obj4 2023-09-30T00:28:03.255 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.255 DEBUG:boto:Path: /dmuhsu-80/obj4 2023-09-30T00:28:03.255 DEBUG:boto:Data: 2023-09-30T00:28:03.255 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.255 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.255 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.255 DEBUG:boto:Params: {} 2023-09-30T00:28:03.255 DEBUG:boto:Token: None 2023-09-30T00:28:03.256 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj4 2023-09-30T00:28:03.256 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MJOBCexsJWdB/8hi7YULmWK+viA= 2023-09-30T00:28:03.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MJOBCexsJWdB/8hi7YULmWK+viA=', 'Content-Length': '0'} 2023-09-30T00:28:03.257 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000468917350a822344-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.257 DEBUG:boto:path=/dmuhsu-80/obj5 2023-09-30T00:28:03.257 DEBUG:boto:auth_path=/dmuhsu-80/obj5 2023-09-30T00:28:03.258 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.258 DEBUG:boto:Path: /dmuhsu-80/obj5 2023-09-30T00:28:03.258 DEBUG:boto:Data: 2023-09-30T00:28:03.258 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.258 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.258 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.258 DEBUG:boto:Params: {} 2023-09-30T00:28:03.258 DEBUG:boto:Token: None 2023-09-30T00:28:03.258 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj5 2023-09-30T00:28:03.259 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QjtKIIayc312CdKUaWrmmjgroC0= 2023-09-30T00:28:03.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QjtKIIayc312CdKUaWrmmjgroC0=', 'Content-Length': '0'} 2023-09-30T00:28:03.260 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005e504e80db641ba0-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.260 DEBUG:boto:path=/dmuhsu-80/obj6 2023-09-30T00:28:03.260 DEBUG:boto:auth_path=/dmuhsu-80/obj6 2023-09-30T00:28:03.261 DEBUG:boto:Method: HEAD 2023-09-30T00:28:03.261 DEBUG:boto:Path: /dmuhsu-80/obj6 2023-09-30T00:28:03.261 DEBUG:boto:Data: 2023-09-30T00:28:03.261 DEBUG:boto:Headers: {} 2023-09-30T00:28:03.261 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:28:03.261 DEBUG:boto:Port: 8001 2023-09-30T00:28:03.261 DEBUG:boto:Params: {} 2023-09-30T00:28:03.261 DEBUG:boto:Token: None 2023-09-30T00:28:03.261 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:03 GMT /dmuhsu-80/obj6 2023-09-30T00:28:03.262 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:T8TKUdJuLNKhm2Em6cGj/kbQV5I= 2023-09-30T00:28:03.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:03 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:T8TKUdJuLNKhm2Em6cGj/kbQV5I=', 'Content-Length': '0'} 2023-09-30T00:28:03.263 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000067188c8bdd05b4d0-0065176b93-234d-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:28:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:03.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.264 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:03.308 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.309 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.394 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:28:03.394 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... ok 2023-09-30T00:28:03.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.396 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:03.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.441 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.480 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:28:03.493 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:28:03.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:25:38.862705Z" 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.495 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:28:03.496 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:03.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:28:03.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.497 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:03.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.584 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:03.584+0000 7f6b70f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:03.585 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:03.585+0000 7f6b70f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:03.600 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:03.600 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:03.600 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:03.601 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:03.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:03.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:03.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:03.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:03.607 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:03Z" 2023-09-30T00:28:03.607 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:03.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:03Z" } 2023-09-30T00:28:03.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:03.608 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:03.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:03.608 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:28:03.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:28:03.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:03.609 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.654 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:28:03.710+0000 7f6742813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:03.711 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:28:03.711+0000 7f6742813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:03.738 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:28:03.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:03.740 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:28:03.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.744 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:28:03.745 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:28:03.745 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:28:03.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:03.745 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:28:03Z" 2023-09-30T00:28:03.745 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:28:03.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:03Z" } 2023-09-30T00:28:03.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:03.746 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:03.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:03.746 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:28:03.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.747 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:03.788 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.789 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.859 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:28:03.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.860 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:03.946 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:03.946 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:03.997 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:28:03.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:03.998 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:04.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:04.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:04.098 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:28:04.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.099 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:04.147 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:04.147 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:04.210 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:28:04.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.211 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:04.256 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:04.256 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:04.308 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:28:04.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.309 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:04.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:04.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:04.414 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 13, 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.415 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:28:04.416 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:28:04.417 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:28:04.419 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:28:04.420 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:28:04.421 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.422 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:28:04.423 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: "directional": [ 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: "source_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:28:04.424 INFO:teuthology.orchestra.run.smithi175.stdout: "dest_zone": "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:28:04.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.430 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.430 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:28:04.431 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:28:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.435 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:28:04.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:28:04.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:28:04.438 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:28:04.439 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:28:04.440 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:28:04.440 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:28:04.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:28:04.441 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:28:04.487 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:04.487 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:04.595 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:28:04.596 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-81 2023-09-30T00:28:04.596 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:28:04.596 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:28:04.596 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.597 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:28:04.597 DEBUG:boto:Data: 2023-09-30T00:28:04.597 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.597 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.597 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.597 DEBUG:boto:Params: {} 2023-09-30T00:28:04.597 DEBUG:boto:Token: None 2023-09-30T00:28:04.597 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/ 2023-09-30T00:28:04.597 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qh6CK/mbWxnpZosCLcZrMLAiDD0= 2023-09-30T00:28:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qh6CK/mbWxnpZosCLcZrMLAiDD0=', 'Content-Length': '0'} 2023-09-30T00:28:04.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ff28eb2bb24f64a-0065176b94-23eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.678 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:28:04.679 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:28:04.679 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.679 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:28:04.679 DEBUG:boto:Data: 2023-09-30T00:28:04.679 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.679 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.679 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.680 DEBUG:boto:Params: {} 2023-09-30T00:28:04.680 DEBUG:boto:Token: None 2023-09-30T00:28:04.680 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/ 2023-09-30T00:28:04.680 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU= 2023-09-30T00:28:04.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU=', 'Content-Length': '0'} 2023-09-30T00:28:04.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad1e194cf1a87605-0065176b94-23eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.684 DEBUG:boto:path=/dmuhsu-81/obj1 2023-09-30T00:28:04.684 DEBUG:boto:auth_path=/dmuhsu-81/obj1 2023-09-30T00:28:04.685 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.685 DEBUG:boto:Path: /dmuhsu-81/obj1 2023-09-30T00:28:04.685 DEBUG:boto:Data: 2023-09-30T00:28:04.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.685 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.685 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.685 DEBUG:boto:Params: {} 2023-09-30T00:28:04.686 DEBUG:boto:Token: None 2023-09-30T00:28:04.686 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/obj1 2023-09-30T00:28:04.686 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cXnZJsI+NPqH9zbOYkUf7hTjiWU= 2023-09-30T00:28:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cXnZJsI+NPqH9zbOYkUf7hTjiWU='} 2023-09-30T00:28:04.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000715687c28a7d8238-0065176b94-23eb-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.725 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:28:04.725 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:28:04.726 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.726 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:28:04.726 DEBUG:boto:Data: 2023-09-30T00:28:04.726 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.726 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.726 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.726 DEBUG:boto:Params: {} 2023-09-30T00:28:04.726 DEBUG:boto:Token: None 2023-09-30T00:28:04.726 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/ 2023-09-30T00:28:04.727 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU= 2023-09-30T00:28:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU=', 'Content-Length': '0'} 2023-09-30T00:28:04.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f907fa8c918823b0-0065176b94-23eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.731 DEBUG:boto:path=/dmuhsu-81/obj2 2023-09-30T00:28:04.731 DEBUG:boto:auth_path=/dmuhsu-81/obj2 2023-09-30T00:28:04.731 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.731 DEBUG:boto:Path: /dmuhsu-81/obj2 2023-09-30T00:28:04.731 DEBUG:boto:Data: 2023-09-30T00:28:04.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.731 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.731 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.731 DEBUG:boto:Params: {} 2023-09-30T00:28:04.732 DEBUG:boto:Token: None 2023-09-30T00:28:04.732 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/obj2 2023-09-30T00:28:04.732 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:TftEAAcY1l/CqaMSSnXMJx45HRA= 2023-09-30T00:28:04.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TftEAAcY1l/CqaMSSnXMJx45HRA='} 2023-09-30T00:28:04.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5d5745f0a43ac49-0065176b94-23eb-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.779 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:28:04.779 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:28:04.779 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.779 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:28:04.779 DEBUG:boto:Data: 2023-09-30T00:28:04.779 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.780 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.780 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.780 DEBUG:boto:Params: {} 2023-09-30T00:28:04.780 DEBUG:boto:Token: None 2023-09-30T00:28:04.780 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/ 2023-09-30T00:28:04.780 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU= 2023-09-30T00:28:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WgADqV7CnxnIyfrcHDAMmDUD8bU=', 'Content-Length': '0'} 2023-09-30T00:28:04.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008b2238d2bafb853-0065176b94-23eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.784 DEBUG:boto:path=/dmuhsu-81/obj3 2023-09-30T00:28:04.784 DEBUG:boto:auth_path=/dmuhsu-81/obj3 2023-09-30T00:28:04.784 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.784 DEBUG:boto:Path: /dmuhsu-81/obj3 2023-09-30T00:28:04.784 DEBUG:boto:Data: 2023-09-30T00:28:04.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.785 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:28:04.785 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.785 DEBUG:boto:Params: {} 2023-09-30T00:28:04.785 DEBUG:boto:Token: None 2023-09-30T00:28:04.785 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-81/obj3 2023-09-30T00:28:04.785 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:gGQt1K025Acgoldanxb2bDeaAns= 2023-09-30T00:28:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:gGQt1K025Acgoldanxb2bDeaAns='} 2023-09-30T00:28:04.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e12912ae9a726b08-0065176b94-23eb-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.810 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=dmuhsu-82 2023-09-30T00:28:04.810 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:28:04.810 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:28:04.811 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.811 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:28:04.811 DEBUG:boto:Data: 2023-09-30T00:28:04.811 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.811 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.811 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.811 DEBUG:boto:Params: {} 2023-09-30T00:28:04.811 DEBUG:boto:Token: None 2023-09-30T00:28:04.812 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/ 2023-09-30T00:28:04.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uFb2Vd8kJoBe09ETERAU0dwojCU= 2023-09-30T00:28:04.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uFb2Vd8kJoBe09ETERAU0dwojCU=', 'Content-Length': '0'} 2023-09-30T00:28:04.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dce2c0bbe22f5d67-0065176b94-1c0b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.883 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:28:04.883 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:28:04.883 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.883 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:28:04.883 DEBUG:boto:Data: 2023-09-30T00:28:04.883 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.883 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.883 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.883 DEBUG:boto:Params: {} 2023-09-30T00:28:04.884 DEBUG:boto:Token: None 2023-09-30T00:28:04.884 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/ 2023-09-30T00:28:04.884 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y= 2023-09-30T00:28:04.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y=', 'Content-Length': '0'} 2023-09-30T00:28:04.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b343953366e54074-0065176b94-1c0b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.887 DEBUG:boto:path=/dmuhsu-82/obj4 2023-09-30T00:28:04.887 DEBUG:boto:auth_path=/dmuhsu-82/obj4 2023-09-30T00:28:04.887 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.887 DEBUG:boto:Path: /dmuhsu-82/obj4 2023-09-30T00:28:04.887 DEBUG:boto:Data: 2023-09-30T00:28:04.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.887 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.887 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.888 DEBUG:boto:Params: {} 2023-09-30T00:28:04.888 DEBUG:boto:Token: None 2023-09-30T00:28:04.888 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/obj4 2023-09-30T00:28:04.888 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:u/1prntWiFiMQurw95F84JTkhAQ= 2023-09-30T00:28:04.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u/1prntWiFiMQurw95F84JTkhAQ='} 2023-09-30T00:28:04.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f1a16cef7373872-0065176b94-1c0b-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.931 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:28:04.931 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:28:04.931 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.931 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:28:04.931 DEBUG:boto:Data: 2023-09-30T00:28:04.931 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.932 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.932 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.932 DEBUG:boto:Params: {} 2023-09-30T00:28:04.932 DEBUG:boto:Token: None 2023-09-30T00:28:04.932 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/ 2023-09-30T00:28:04.932 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y= 2023-09-30T00:28:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y=', 'Content-Length': '0'} 2023-09-30T00:28:04.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080395560e7455e75-0065176b94-1c0b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.935 DEBUG:boto:path=/dmuhsu-82/obj5 2023-09-30T00:28:04.935 DEBUG:boto:auth_path=/dmuhsu-82/obj5 2023-09-30T00:28:04.935 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.935 DEBUG:boto:Path: /dmuhsu-82/obj5 2023-09-30T00:28:04.935 DEBUG:boto:Data: 2023-09-30T00:28:04.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.936 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.936 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.936 DEBUG:boto:Params: {} 2023-09-30T00:28:04.936 DEBUG:boto:Token: None 2023-09-30T00:28:04.936 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/obj5 2023-09-30T00:28:04.936 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/xPRL0yzqJb5EUE1+qARhj/TzQE= 2023-09-30T00:28:04.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/xPRL0yzqJb5EUE1+qARhj/TzQE='} 2023-09-30T00:28:04.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000630f224360e9de37-0065176b94-1c0b-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.945 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:28:04.945 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:28:04.945 DEBUG:boto:Method: HEAD 2023-09-30T00:28:04.946 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:28:04.946 DEBUG:boto:Data: 2023-09-30T00:28:04.946 DEBUG:boto:Headers: {} 2023-09-30T00:28:04.946 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.946 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.946 DEBUG:boto:Params: {} 2023-09-30T00:28:04.946 DEBUG:boto:Token: None 2023-09-30T00:28:04.946 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/ 2023-09-30T00:28:04.946 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y= 2023-09-30T00:28:04.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:thUyarOiNzKlp+6Ui7ffy1UYS4Y=', 'Content-Length': '0'} 2023-09-30T00:28:04.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ab36709859708e0-0065176b94-1c0b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.949 DEBUG:boto:path=/dmuhsu-82/obj6 2023-09-30T00:28:04.949 DEBUG:boto:auth_path=/dmuhsu-82/obj6 2023-09-30T00:28:04.949 DEBUG:boto:Method: PUT 2023-09-30T00:28:04.949 DEBUG:boto:Path: /dmuhsu-82/obj6 2023-09-30T00:28:04.949 DEBUG:boto:Data: 2023-09-30T00:28:04.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:28:04.950 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:28:04.950 DEBUG:boto:Port: 8000 2023-09-30T00:28:04.950 DEBUG:boto:Params: {} 2023-09-30T00:28:04.950 DEBUG:boto:Token: None 2023-09-30T00:28:04.950 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:28:04 GMT /dmuhsu-82/obj6 2023-09-30T00:28:04.950 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rqovPR03xgJIcbhksY7Yc8ssrck= 2023-09-30T00:28:04.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:28:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rqovPR03xgJIcbhksY7Yc8ssrck='} 2023-09-30T00:28:04.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ce496e7d34ec389-0065176b94-1c0b-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:28:04.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:04.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:04.999 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:05.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:05.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:05.144 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:05.181 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:28:04.864312Z" 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:28:05.182 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:28:05.183 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:05.183 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:28:05.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:28:05.183 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:28:05.184 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:05.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:28:05.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:05.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:05.185 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:05.230 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:05.230 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:05.263 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:05.261+0000 7fa7e0727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:05.263 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:05.262+0000 7fa7e0727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:05.276 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:05.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:05.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:05.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:05.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:05.282 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:05.283 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:05Z" 2023-09-30T00:28:05.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:05.285 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:05Z" } 2023-09-30T00:28:05.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:05.285 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:05.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:05.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:10.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:10.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:10.287 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:10.340 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:10.340 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:10.373 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:10.372+0000 7ff6a6d2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:10.374 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:10.373+0000 7ff6a6d2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:10.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:10.388 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:10.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:10.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:10.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:10.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:10.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:10.394 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:10.395 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:10Z" 2023-09-30T00:28:10.395 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:10.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:10Z" } 2023-09-30T00:28:10.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:10.396 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:10.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:10.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:15.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:15.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:15.398 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:15.450 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:15.451 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:15.484 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:15.483+0000 7f518968b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:15.485 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:15.484+0000 7f518968b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:15.499 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.500 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:15.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:15.502 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:15.504 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:15.505 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:15.506 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:15Z" 2023-09-30T00:28:15.507 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:15.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:15Z" } 2023-09-30T00:28:15.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:15.511 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:15.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:15.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:20.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:20.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:20.513 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:20.566 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:20.566 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:20.601 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:20.600+0000 7f7af92c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:20.601 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:20.601+0000 7f7af92c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:20.618 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:20.619 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:20.620 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:20.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:20.622 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:20.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.624 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:20.625 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:20.625 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:20.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:20.625 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:20Z" 2023-09-30T00:28:20.625 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:20.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:20Z" } 2023-09-30T00:28:20.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:20.626 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:20.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:20.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:25.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:25.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:25.628 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:25.681 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:25.681 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:25.715 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:25.714+0000 7fbcaad33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:25.715 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:25.715+0000 7fbcaad33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:25.730 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:25.731 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:25.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:25.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:25.734 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:25.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:25.736 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:25.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:25.737 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:25Z" 2023-09-30T00:28:25.737 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:25.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:25Z" } 2023-09-30T00:28:25.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:25.738 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:25.738 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:25.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:30.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:30.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:30.739 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:30.792 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:30.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:30.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:30.825+0000 7ff05e126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:30.827 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:30.826+0000 7ff05e126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:30.842 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:30.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:30.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:30.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:30.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:30.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:30Z" 2023-09-30T00:28:30.848 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:30.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:30Z" } 2023-09-30T00:28:30.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:30.850 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:30.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:30.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:35.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:35.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:35.852 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:35.905 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:35.905 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:35.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:35.939+0000 7f3b4cf27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:35.940 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:35.940+0000 7f3b4cf27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:35.955 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:35.955 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:35.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:35.957 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:35.958 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:35.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:35Z" 2023-09-30T00:28:35.961 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:35.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:35Z" } 2023-09-30T00:28:35.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:35.963 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:35.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:35.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:40.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:40.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:40.965 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:41.019 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:41.020 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:41.053 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:41.052+0000 7f7fd9eb8880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:41.054 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:41.053+0000 7f7fd9eb8880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:41.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:41.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:41.071 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:41.073 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:41.074 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:41.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:41.076 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:41.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:41.076 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:41Z" 2023-09-30T00:28:41.076 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:41.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:41Z" } 2023-09-30T00:28:41.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:41.077 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:41.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:41.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:46.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:46.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:46.079 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:46.132 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:46.132 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:46.165 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:46.164+0000 7f062c94b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:46.166 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:46.165+0000 7f062c94b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:46.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:46.182 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:46.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:46.184 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:46.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:46.187 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:46.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:46.188 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:46Z" 2023-09-30T00:28:46.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:46.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:46Z" } 2023-09-30T00:28:46.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:46.189 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:46.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:46.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:51.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:51.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:51.191 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:51.241 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:51.242 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:51.274 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:51.273+0000 7fba931a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:51.274 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:51.274+0000 7fba931a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:51.289 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:51.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:51.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:51.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:51.294 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:51.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:51.296 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:51Z" 2023-09-30T00:28:51.296 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:51.297 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:51Z" } 2023-09-30T00:28:51.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:51.297 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:51.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:51.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:28:56.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:56.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:28:56.299 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:28:56.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:28:56.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:28:56.386 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:56.386+0000 7f3e93e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:56.387 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:28:56.387+0000 7f3e93e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:28:56.402 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:28:56.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:56.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:56.405 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:56.406 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:28:56.407 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:28:56.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:28:56Z" 2023-09-30T00:28:56.409 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:28:56.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:28:56Z" } 2023-09-30T00:28:56.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:28:56.410 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:56.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:28:56.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:01.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:01.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:01.412 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:01.465 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:01.465 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:01.498 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:01.497+0000 7fd135a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:01.499 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:01.498+0000 7fd135a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:01.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:01.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:01.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:01.517 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:01.518 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.519 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:01.520 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:01.520 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:01.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:01.520 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:01Z" 2023-09-30T00:29:01.520 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:01.521 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:01Z" } 2023-09-30T00:29:01.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:01.521 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:01.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:01.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:06.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:06.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:06.523 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:06.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:06.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:06.612 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:06.611+0000 7f10c7527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:06.613 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:06.612+0000 7f10c7527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:06.627 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:06.627 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:06.627 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:06.628 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:06.629 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:06.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:06.631 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:06.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:06.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:06.634 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:06.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:06.634 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:06Z" 2023-09-30T00:29:06.634 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:06.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:06Z" } 2023-09-30T00:29:06.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:06.635 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:06.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:06.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:11.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:11.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:11.637 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:11.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:11.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:11.724 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:11.723+0000 7feaf6f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:11.724 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:11.724+0000 7feaf6f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:11Z" 2023-09-30T00:29:11.747 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:11.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:11Z" } 2023-09-30T00:29:11.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:11.748 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:11.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:11.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:16.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:16.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:16.750 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:16.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:16.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:16.837 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:16.836+0000 7f91c7f75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:16.837 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:16.837+0000 7f91c7f75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:16.851 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.852 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:16.853 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:16.854 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:16.855 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:16.856 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:16Z" 2023-09-30T00:29:16.858 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:16.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:16Z" } 2023-09-30T00:29:16.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:16.860 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:16.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:16.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:21.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:21.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:21.861 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:21.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:21.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:21.948 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:21.948+0000 7f42c4826880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:21.948 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:21.949+0000 7f42c4826880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:21.963 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:21.963 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:21.963 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:21.963 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:21.964 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:21.965 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:21.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:21.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:21.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:21.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:21.970 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:21Z" 2023-09-30T00:29:21.970 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:21.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:21Z" } 2023-09-30T00:29:21.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:21.971 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:21.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:21.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:26.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:26.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:26.972 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:27.026 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:27.026 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:27.059 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:27.059+0000 7f53d0d33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:27.060 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:29:27.060+0000 7f53d0d33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:27.075 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:27.075 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:27.075 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:27.076 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:27.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:27.079 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:27.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:29:27.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:27.082 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:27Z" 2023-09-30T00:29:27.082 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:27.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:27Z" } 2023-09-30T00:29:27.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:27.083 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:27.083 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:27.083 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:29:27.083 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:29:27.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:27.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:27.084 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:27.137 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:27.137 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:27.190 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:27.189+0000 7efc5632b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:27.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:27.190+0000 7efc5632b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:27.206 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:29:27.206 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:29:27.206 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:27.207 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:27.208 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:27.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:27.210 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:27.211 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.212 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:29:27.213 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:29:27.213 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:29:27.213 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:27.213 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:29:27Z" 2023-09-30T00:29:27.213 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:29:27.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:27Z" } 2023-09-30T00:29:27.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:27.214 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:27.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:27.215 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:32.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:32.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:32.216 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:32.270 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:32.270 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:32.306 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:32.304+0000 7f08277a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:32.307 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:32.305+0000 7f08277a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:32.324 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:29:32.325 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:32.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:32.327 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:32.328 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:32.329 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:32.330 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:29:32Z" 2023-09-30T00:29:32.331 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:29:32.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:32Z" } 2023-09-30T00:29:32.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:32.333 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:32.333 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:32.333 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:37.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:37.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:37.335 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:37.387 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:37.388 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:37.430 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:37.429+0000 7fe943e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:37.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:37.430+0000 7fe943e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:37.447 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:29:37.448 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:37.449 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:29:37.450 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033538.862705_872.1", 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:25:38.862705Z", 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.451 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:37.452 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:29:37Z" 2023-09-30T00:29:37.453 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:29:37.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033538.862705_872.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:25:38.862705Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:37Z" } 2023-09-30T00:29:37.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033538.862705_872.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:25:38.862705Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:37.455 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:37.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033538.862705_872.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:37.455 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033684.864312_917.1 target=1_1696033538.862705_872.1 2023-09-30T00:29:42.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.457 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:42.509 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:42.509 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:42.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:42.550+0000 7f9359320880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:42.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:29:42.551+0000 7f9359320880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:29:42.569 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.570 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:29:42.571 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:42.572 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:29:42.573 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:29:42.574 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:29:42.575 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:29:42Z" 2023-09-30T00:29:42.576 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:29:42.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:29:42Z" } 2023-09-30T00:29:42.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:29:42.577 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:42.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:29:42.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:29:42.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.578 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:42.623 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:42.623 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:42.699 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:29:42.699 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.699 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:29:42.699 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:29:42.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:29:42.701 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:29:42.701 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:29:42.701 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:29:42.701 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:29:42.702 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:29:42.702 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:29:42.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:42.702 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:42.757 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:42.757 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:42.807 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:42.808 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:42.809 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023247", 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.759523Z" 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:42.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:42.811 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:42.812 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018054", 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.719574Z" 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:29:42.813 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:29:42.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:42.814 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:42Z" 2023-09-30T00:29:42.814 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:42.817 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023247", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:27:39.759523Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018054", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.719574Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:29:42Z" } 2023-09-30T00:29:42.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023247', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.759523Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018054', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.719574Z'}}] 2023-09-30T00:29:42.817 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:29:42.818 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023247', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018054'} 2023-09-30T00:29:42.818 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023527 target=00000000000000000000:00000000000000023247, shard=3 master=00000000000000000000:00000000000000018194 target=00000000000000000000:00000000000000018054 2023-09-30T00:29:47.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:47.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:47.819 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:47.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:47.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:47.926 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023247", 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.759523Z" 2023-09-30T00:29:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:29:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018054", 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.719574Z" 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:29:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:29:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:47Z" 2023-09-30T00:29:47.933 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:47.934 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023247", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:27:39.759523Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018054", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.719574Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:29:47Z" } 2023-09-30T00:29:47.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023247', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.759523Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018054', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.719574Z'}}] 2023-09-30T00:29:47.934 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:29:47.934 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023247', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018054'} 2023-09-30T00:29:47.935 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023527 target=00000000000000000000:00000000000000023247, shard=3 master=00000000000000000000:00000000000000018194 target=00000000000000000000:00000000000000018054 2023-09-30T00:29:52.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:52.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:52.936 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:52.991 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:52.991 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.041 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:29:53.042 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023247", 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:29:53.043 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.759523Z" 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:53.044 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018054", 2023-09-30T00:29:53.045 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.719574Z" 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:53.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:29:53.047 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:29:53.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:53.047 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:53Z" 2023-09-30T00:29:53.047 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:53.048 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023247", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:27:39.759523Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018054", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.719574Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:29:53Z" } 2023-09-30T00:29:53.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023247', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.759523Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018054', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.719574Z'}}] 2023-09-30T00:29:53.049 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:29:53.049 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023247', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018054'} 2023-09-30T00:29:53.049 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023527 target=00000000000000000000:00000000000000023247, shard=3 master=00000000000000000000:00000000000000018194 target=00000000000000000000:00000000000000018054 2023-09-30T00:29:58.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:58.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:29:58.051 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:29:58.106 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:29:58.106 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:29:58.156 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:58.157 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023247", 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.759523Z" 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:58.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:29:58.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018054", 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:29:58.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.719574Z" 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:29:58Z" 2023-09-30T00:29:58.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:29:58.163 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023247", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:27:39.759523Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018054", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.719574Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:29:58Z" } 2023-09-30T00:29:58.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023247', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.759523Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018054', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.719574Z'}}] 2023-09-30T00:29:58.164 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:29:58.164 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023247', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018054'} 2023-09-30T00:29:58.164 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023527 target=00000000000000000000:00000000000000023247, shard=3 master=00000000000000000000:00000000000000018194 target=00000000000000000000:00000000000000018054 2023-09-30T00:30:03.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:03.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:03.166 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:03.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:03.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.271 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:30:03.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023247", 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.759523Z" 2023-09-30T00:30:03.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:03.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018054", 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:03.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.719574Z" 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:03Z" 2023-09-30T00:30:03.277 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:03.277 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023247", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:27:39.759523Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018054", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:27:39.719574Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:30:03Z" } 2023-09-30T00:30:03.278 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023247', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.759523Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018054', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.719574Z'}}] 2023-09-30T00:30:03.278 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:30:03.278 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023247', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018054'} 2023-09-30T00:30:03.278 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023527 target=00000000000000000000:00000000000000023247, shard=3 master=00000000000000000000:00000000000000018194 target=00000000000000000000:00000000000000018054 2023-09-30T00:30:08.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:08.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:08.280 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:08.334 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:08.334 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:08.384 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:30:08.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:30:08.386 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:08.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:30:08.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:30:08.389 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:08.390 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:08.391 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:08Z" 2023-09-30T00:30:08.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:08.393 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:30:08Z" } 2023-09-30T00:30:08.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:30:08.393 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:30:08.393 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:30:08.394 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:30:08.394 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:30:08.394 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:30:08.394 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.395 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:30:08.395 DEBUG:boto:Data: 2023-09-30T00:30:08.395 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.395 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.395 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.395 DEBUG:boto:Params: {} 2023-09-30T00:30:08.396 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:30:08.396 DEBUG:boto:Token: None 2023-09-30T00:30:08.396 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/ 2023-09-30T00:30:08.397 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NJk/uOPUrDTbj3UMx0YPN6WjfMQ= 2023-09-30T00:30:08.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NJk/uOPUrDTbj3UMx0YPN6WjfMQ=', 'Content-Length': '0'} 2023-09-30T00:30:08.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052f76a7e17049e1e-0065176c10-1c0b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.402 DEBUG:boto:path=/dmuhsu-81/obj1 2023-09-30T00:30:08.402 DEBUG:boto:auth_path=/dmuhsu-81/obj1 2023-09-30T00:30:08.402 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.403 DEBUG:boto:Path: /dmuhsu-81/obj1 2023-09-30T00:30:08.403 DEBUG:boto:Data: 2023-09-30T00:30:08.403 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.403 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.403 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.404 DEBUG:boto:Params: {} 2023-09-30T00:30:08.404 DEBUG:boto:Token: None 2023-09-30T00:30:08.404 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj1 2023-09-30T00:30:08.404 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DLfGvgI5Q40jtOimi0rsh+fmeyo= 2023-09-30T00:30:08.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DLfGvgI5Q40jtOimi0rsh+fmeyo=', 'Content-Length': '0'} 2023-09-30T00:30:08.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094f733af0ee9e1ed-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.407 DEBUG:boto:path=/dmuhsu-81/obj1 2023-09-30T00:30:08.407 DEBUG:boto:auth_path=/dmuhsu-81/obj1 2023-09-30T00:30:08.407 DEBUG:boto:Method: GET 2023-09-30T00:30:08.408 DEBUG:boto:Path: /dmuhsu-81/obj1 2023-09-30T00:30:08.408 DEBUG:boto:Data: 2023-09-30T00:30:08.408 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.408 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.408 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.409 DEBUG:boto:Params: {} 2023-09-30T00:30:08.409 DEBUG:boto:Token: None 2023-09-30T00:30:08.409 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj1 2023-09-30T00:30:08.409 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UCpsWUXs8CbDVUqwIxYLXTNwzUY= 2023-09-30T00:30:08.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UCpsWUXs8CbDVUqwIxYLXTNwzUY=', 'Content-Length': '0'} 2023-09-30T00:30:08.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b5149cc3e43bac1-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.412 DEBUG:boto:path=/dmuhsu-81/obj2 2023-09-30T00:30:08.412 DEBUG:boto:auth_path=/dmuhsu-81/obj2 2023-09-30T00:30:08.412 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.412 DEBUG:boto:Path: /dmuhsu-81/obj2 2023-09-30T00:30:08.413 DEBUG:boto:Data: 2023-09-30T00:30:08.413 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.413 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.413 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.413 DEBUG:boto:Params: {} 2023-09-30T00:30:08.414 DEBUG:boto:Token: None 2023-09-30T00:30:08.414 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj2 2023-09-30T00:30:08.414 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Rh6vdQcCgfijUevxG8PnaU59bk4= 2023-09-30T00:30:08.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Rh6vdQcCgfijUevxG8PnaU59bk4=', 'Content-Length': '0'} 2023-09-30T00:30:08.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007370edada8463b4e-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.417 DEBUG:boto:path=/dmuhsu-81/obj2 2023-09-30T00:30:08.417 DEBUG:boto:auth_path=/dmuhsu-81/obj2 2023-09-30T00:30:08.417 DEBUG:boto:Method: GET 2023-09-30T00:30:08.417 DEBUG:boto:Path: /dmuhsu-81/obj2 2023-09-30T00:30:08.417 DEBUG:boto:Data: 2023-09-30T00:30:08.417 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.417 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.418 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.418 DEBUG:boto:Params: {} 2023-09-30T00:30:08.418 DEBUG:boto:Token: None 2023-09-30T00:30:08.418 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj2 2023-09-30T00:30:08.418 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:rFvXvyxRZLnRCRm76iEV5HIZ5RE= 2023-09-30T00:30:08.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rFvXvyxRZLnRCRm76iEV5HIZ5RE=', 'Content-Length': '0'} 2023-09-30T00:30:08.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000718385079d190758-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.420 DEBUG:boto:path=/dmuhsu-81/obj3 2023-09-30T00:30:08.420 DEBUG:boto:auth_path=/dmuhsu-81/obj3 2023-09-30T00:30:08.420 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.421 DEBUG:boto:Path: /dmuhsu-81/obj3 2023-09-30T00:30:08.421 DEBUG:boto:Data: 2023-09-30T00:30:08.421 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.421 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.421 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.421 DEBUG:boto:Params: {} 2023-09-30T00:30:08.421 DEBUG:boto:Token: None 2023-09-30T00:30:08.421 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj3 2023-09-30T00:30:08.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VP8k7HVZ2i/zA5renHQD1dsY5lU= 2023-09-30T00:30:08.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VP8k7HVZ2i/zA5renHQD1dsY5lU=', 'Content-Length': '0'} 2023-09-30T00:30:08.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7bbc712b3290e25-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.423 DEBUG:boto:path=/dmuhsu-81/obj3 2023-09-30T00:30:08.423 DEBUG:boto:auth_path=/dmuhsu-81/obj3 2023-09-30T00:30:08.423 DEBUG:boto:Method: GET 2023-09-30T00:30:08.423 DEBUG:boto:Path: /dmuhsu-81/obj3 2023-09-30T00:30:08.424 DEBUG:boto:Data: 2023-09-30T00:30:08.424 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.424 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:30:08.424 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.424 DEBUG:boto:Params: {} 2023-09-30T00:30:08.424 DEBUG:boto:Token: None 2023-09-30T00:30:08.424 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj3 2023-09-30T00:30:08.424 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QvuEDgftyOqlh/T67nhwbM2l+GU= 2023-09-30T00:30:08.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QvuEDgftyOqlh/T67nhwbM2l+GU=', 'Content-Length': '0'} 2023-09-30T00:30:08.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:28:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dad4db522dc60e4d-0065176c10-1c0b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.470 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:30:08.471 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:30:08.471 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.471 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:30:08.471 DEBUG:boto:Data: 2023-09-30T00:30:08.471 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.471 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.471 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.471 DEBUG:boto:Params: {} 2023-09-30T00:30:08.473 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:30:08.473 DEBUG:boto:Token: None 2023-09-30T00:30:08.473 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/ 2023-09-30T00:30:08.473 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:NJk/uOPUrDTbj3UMx0YPN6WjfMQ= 2023-09-30T00:30:08.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NJk/uOPUrDTbj3UMx0YPN6WjfMQ=', 'Content-Length': '0'} 2023-09-30T00:30:08.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081e8c08a27d096af-0065176c10-23f5-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.479 DEBUG:boto:path=/dmuhsu-81/obj1 2023-09-30T00:30:08.479 DEBUG:boto:auth_path=/dmuhsu-81/obj1 2023-09-30T00:30:08.479 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.479 DEBUG:boto:Path: /dmuhsu-81/obj1 2023-09-30T00:30:08.479 DEBUG:boto:Data: 2023-09-30T00:30:08.479 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.480 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.480 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.480 DEBUG:boto:Params: {} 2023-09-30T00:30:08.480 DEBUG:boto:Token: None 2023-09-30T00:30:08.480 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj1 2023-09-30T00:30:08.480 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DLfGvgI5Q40jtOimi0rsh+fmeyo= 2023-09-30T00:30:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DLfGvgI5Q40jtOimi0rsh+fmeyo=', 'Content-Length': '0'} 2023-09-30T00:30:08.482 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000078862812fd4b125c-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.482 DEBUG:boto:path=/dmuhsu-81/obj2 2023-09-30T00:30:08.482 DEBUG:boto:auth_path=/dmuhsu-81/obj2 2023-09-30T00:30:08.482 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.482 DEBUG:boto:Path: /dmuhsu-81/obj2 2023-09-30T00:30:08.482 DEBUG:boto:Data: 2023-09-30T00:30:08.482 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.483 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.483 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.483 DEBUG:boto:Params: {} 2023-09-30T00:30:08.483 DEBUG:boto:Token: None 2023-09-30T00:30:08.483 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj2 2023-09-30T00:30:08.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Rh6vdQcCgfijUevxG8PnaU59bk4= 2023-09-30T00:30:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Rh6vdQcCgfijUevxG8PnaU59bk4=', 'Content-Length': '0'} 2023-09-30T00:30:08.485 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b06c2ee1db45f57b-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.485 DEBUG:boto:path=/dmuhsu-81/obj3 2023-09-30T00:30:08.485 DEBUG:boto:auth_path=/dmuhsu-81/obj3 2023-09-30T00:30:08.485 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.485 DEBUG:boto:Path: /dmuhsu-81/obj3 2023-09-30T00:30:08.485 DEBUG:boto:Data: 2023-09-30T00:30:08.485 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.485 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.486 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.486 DEBUG:boto:Params: {} 2023-09-30T00:30:08.486 DEBUG:boto:Token: None 2023-09-30T00:30:08.486 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-81/obj3 2023-09-30T00:30:08.486 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:VP8k7HVZ2i/zA5renHQD1dsY5lU= 2023-09-30T00:30:08.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VP8k7HVZ2i/zA5renHQD1dsY5lU=', 'Content-Length': '0'} 2023-09-30T00:30:08.487 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a748584ee01c6af7-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.488 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:30:08.488 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:30:08.488 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.488 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:30:08.488 DEBUG:boto:Data: 2023-09-30T00:30:08.488 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.488 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:30:08.488 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.488 DEBUG:boto:Params: {} 2023-09-30T00:30:08.489 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:30:08.489 DEBUG:boto:Token: None 2023-09-30T00:30:08.489 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/ 2023-09-30T00:30:08.489 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YvRupOpftm79FSHkHVnXzKHS2zE= 2023-09-30T00:30:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YvRupOpftm79FSHkHVnXzKHS2zE=', 'Content-Length': '0'} 2023-09-30T00:30:08.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031275adad8df0a05-0065176c10-23eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.493 DEBUG:boto:path=/dmuhsu-82/obj4 2023-09-30T00:30:08.493 DEBUG:boto:auth_path=/dmuhsu-82/obj4 2023-09-30T00:30:08.493 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.493 DEBUG:boto:Path: /dmuhsu-82/obj4 2023-09-30T00:30:08.493 DEBUG:boto:Data: 2023-09-30T00:30:08.493 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.493 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:30:08.494 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.494 DEBUG:boto:Params: {} 2023-09-30T00:30:08.494 DEBUG:boto:Token: None 2023-09-30T00:30:08.494 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj4 2023-09-30T00:30:08.494 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GViyycy97ZqRkqP0IkTGVRFC8tI= 2023-09-30T00:30:08.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GViyycy97ZqRkqP0IkTGVRFC8tI=', 'Content-Length': '0'} 2023-09-30T00:30:08.496 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005d9774228e36e1b7-0065176c10-23eb-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.496 DEBUG:boto:path=/dmuhsu-82/obj5 2023-09-30T00:30:08.496 DEBUG:boto:auth_path=/dmuhsu-82/obj5 2023-09-30T00:30:08.496 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.496 DEBUG:boto:Path: /dmuhsu-82/obj5 2023-09-30T00:30:08.496 DEBUG:boto:Data: 2023-09-30T00:30:08.496 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.497 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:30:08.497 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.497 DEBUG:boto:Params: {} 2023-09-30T00:30:08.497 DEBUG:boto:Token: None 2023-09-30T00:30:08.497 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj5 2023-09-30T00:30:08.497 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Yiycukve+EcA5WH8l3fN/6BIp+o= 2023-09-30T00:30:08.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Yiycukve+EcA5WH8l3fN/6BIp+o=', 'Content-Length': '0'} 2023-09-30T00:30:08.499 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001a326f4de1897311-0065176c10-23eb-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.499 DEBUG:boto:path=/dmuhsu-82/obj6 2023-09-30T00:30:08.499 DEBUG:boto:auth_path=/dmuhsu-82/obj6 2023-09-30T00:30:08.499 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.499 DEBUG:boto:Path: /dmuhsu-82/obj6 2023-09-30T00:30:08.499 DEBUG:boto:Data: 2023-09-30T00:30:08.500 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.500 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:30:08.500 DEBUG:boto:Port: 8000 2023-09-30T00:30:08.500 DEBUG:boto:Params: {} 2023-09-30T00:30:08.500 DEBUG:boto:Token: None 2023-09-30T00:30:08.500 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj6 2023-09-30T00:30:08.500 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wf/2o7FBr5f12efzhphTXo3+TCQ= 2023-09-30T00:30:08.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wf/2o7FBr5f12efzhphTXo3+TCQ=', 'Content-Length': '0'} 2023-09-30T00:30:08.502 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000532283b7316213c4-0065176c10-23eb-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.502 DEBUG:boto:path=/dmuhsu-82/ 2023-09-30T00:30:08.502 DEBUG:boto:auth_path=/dmuhsu-82/ 2023-09-30T00:30:08.502 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.502 DEBUG:boto:Path: /dmuhsu-82/ 2023-09-30T00:30:08.503 DEBUG:boto:Data: 2023-09-30T00:30:08.503 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.503 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.503 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.503 DEBUG:boto:Params: {} 2023-09-30T00:30:08.503 DEBUG:boto:Token: None 2023-09-30T00:30:08.503 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/ 2023-09-30T00:30:08.503 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:YvRupOpftm79FSHkHVnXzKHS2zE= 2023-09-30T00:30:08.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YvRupOpftm79FSHkHVnXzKHS2zE=', 'Content-Length': '0'} 2023-09-30T00:30:08.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3232bcc4797138d-0065176c10-23f5-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.506 DEBUG:boto:path=/dmuhsu-82/obj4 2023-09-30T00:30:08.506 DEBUG:boto:auth_path=/dmuhsu-82/obj4 2023-09-30T00:30:08.506 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.506 DEBUG:boto:Path: /dmuhsu-82/obj4 2023-09-30T00:30:08.506 DEBUG:boto:Data: 2023-09-30T00:30:08.506 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.506 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.507 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.507 DEBUG:boto:Params: {} 2023-09-30T00:30:08.507 DEBUG:boto:Token: None 2023-09-30T00:30:08.507 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj4 2023-09-30T00:30:08.507 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:GViyycy97ZqRkqP0IkTGVRFC8tI= 2023-09-30T00:30:08.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GViyycy97ZqRkqP0IkTGVRFC8tI=', 'Content-Length': '0'} 2023-09-30T00:30:08.509 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004c0d0f972ac21b40-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.509 DEBUG:boto:path=/dmuhsu-82/obj5 2023-09-30T00:30:08.509 DEBUG:boto:auth_path=/dmuhsu-82/obj5 2023-09-30T00:30:08.509 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.509 DEBUG:boto:Path: /dmuhsu-82/obj5 2023-09-30T00:30:08.509 DEBUG:boto:Data: 2023-09-30T00:30:08.509 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.509 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.510 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.510 DEBUG:boto:Params: {} 2023-09-30T00:30:08.510 DEBUG:boto:Token: None 2023-09-30T00:30:08.510 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj5 2023-09-30T00:30:08.510 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Yiycukve+EcA5WH8l3fN/6BIp+o= 2023-09-30T00:30:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Yiycukve+EcA5WH8l3fN/6BIp+o=', 'Content-Length': '0'} 2023-09-30T00:30:08.511 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001ef1db5835281279-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.512 DEBUG:boto:path=/dmuhsu-82/obj6 2023-09-30T00:30:08.512 DEBUG:boto:auth_path=/dmuhsu-82/obj6 2023-09-30T00:30:08.512 DEBUG:boto:Method: HEAD 2023-09-30T00:30:08.512 DEBUG:boto:Path: /dmuhsu-82/obj6 2023-09-30T00:30:08.512 DEBUG:boto:Data: 2023-09-30T00:30:08.512 DEBUG:boto:Headers: {} 2023-09-30T00:30:08.512 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:30:08.512 DEBUG:boto:Port: 8001 2023-09-30T00:30:08.512 DEBUG:boto:Params: {} 2023-09-30T00:30:08.513 DEBUG:boto:Token: None 2023-09-30T00:30:08.513 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:30:08 GMT /dmuhsu-82/obj6 2023-09-30T00:30:08.513 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wf/2o7FBr5f12efzhphTXo3+TCQ= 2023-09-30T00:30:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:30:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wf/2o7FBr5f12efzhphTXo3+TCQ=', 'Content-Length': '0'} 2023-09-30T00:30:08.514 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000180e267131c38ba2-0065176c10-23f5-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:30:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:30:08.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.515 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:08.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:08.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:08.630 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:30:08.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.631 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:08.681 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:08.681 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:08.736 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-30T00:30:08.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.737 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:08.786 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:08.786 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:08.839 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:30:08.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.840 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:08.890 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:08.890 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:08.973 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:30:08.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:08.974 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:09.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:09.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:09.091 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:30:09.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.092 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:09.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:09.136 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:09.214 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:30:09.214 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:09.215 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 14, 2023-09-30T00:30:09.215 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:30:09.215 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:30:09.215 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:30:09.215 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:30:09.216 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:30:09.217 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:30:09.217 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.217 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:30:09.217 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:30:09.217 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:30:09.218 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:30:09.219 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.219 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:30:09.219 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:30:09.219 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:30:09.220 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.221 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:30:09.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:30:09.223 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:30:09.224 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:30:09.224 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:30:09.224 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.224 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:30:09.225 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:30:09.226 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.227 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:30:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:30:09.229 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:30:09.230 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:30:09.231 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.238 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:30:09.238 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.238 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.238 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.238 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:30:09.239 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:30:09.240 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:30:09.241 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:30:09.242 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "allowed" 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.243 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:30:09.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.245 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.246 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:30:09.247 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:30:09.248 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:30:09.249 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:30:09.249 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:30:09.249 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:30:09.249 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:30:09.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:30:09.250 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:30:09.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:30:09.252 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:09.253 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:09.253 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:30:09.253 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:30:09.253 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:30:09.253 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:30:09.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.255 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:09.298 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:09.298 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:09.412 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:30:09.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.413 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-81 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:09.516 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:09.516 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:09.708 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:30:09.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.709 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-81 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:09.790 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:09.790 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:09.982 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:30:09.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-81', '--source-bucket=*', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-81', '--source-bucket=*', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:09.983 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-81 '--source-bucket=*' '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:10.059 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:10.059 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:10.226 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:30:10.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:10.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:10.227 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-81 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:10.281 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:10.281 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:10.414 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:30:10.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:10.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:30:10.415 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dmuhsu-81 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:30:10.462 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:10.462 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:10.586 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:30:10.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:10.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:10.587 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:10.636 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:10.636 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:10.712 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:10.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:30:10.358365Z" 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:30:10.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:30:10.744 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:30:10.744 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:10.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:30:10.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:10.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:10.745 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:10.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:10.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:10.831 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:10.831+0000 7f373dd1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:10.832 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:10.832+0000 7f373dd1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:10.846 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.847 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:10.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:10.849 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:10.850 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:10.851 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.852 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:10.853 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:10.853 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:10.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:10.853 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:10Z" 2023-09-30T00:30:10.853 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:10.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:10Z" } 2023-09-30T00:30:10.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:10.854 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:10.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:10.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:15.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:15.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:15.855 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:15.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:15.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:15.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:15.941+0000 7f59e6624880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:15.942 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:15.942+0000 7f59e6624880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:15.957 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:15.958 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:15.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:15.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:15Z" 2023-09-30T00:30:15.964 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:15.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:15Z" } 2023-09-30T00:30:15.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:15.966 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:15.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:15.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:20.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:20.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:20.967 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:21.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:21.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:21.055 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:21.054+0000 7f26e8c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:21.055 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:21.055+0000 7f26e8c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:21.070 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:21Z" 2023-09-30T00:30:21.076 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:21.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:21Z" } 2023-09-30T00:30:21.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:21.078 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:21.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:21.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:26.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:26.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:26.079 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:26.133 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:26.133 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:26.166 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:26.165+0000 7ff0e6727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:26.167 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:26.166+0000 7ff0e6727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:26.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:26Z" 2023-09-30T00:30:26.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:26.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:26Z" } 2023-09-30T00:30:26.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:26.189 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:26.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:26.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:31.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:31.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:31.191 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:31.245 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:31.246 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:31.277 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:31.277+0000 7fbd08ddc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:31.278 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:31.278+0000 7fbd08ddc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:31.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:31.293 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:31.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:31.295 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:31.296 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:31.297 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:31.298 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:31.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:31.300 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:31Z" 2023-09-30T00:30:31.300 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:31.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:31Z" } 2023-09-30T00:30:31.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:31.301 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:31.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:31.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:36.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:36.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:36.303 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:36.356 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:36.356 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:36.389 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:36.388+0000 7f7490d34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:36.390 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:36.389+0000 7f7490d34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.405 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:36.406 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:36.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:36.408 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:36.409 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:36.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:36.411 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:36.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:36.411 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:36Z" 2023-09-30T00:30:36.411 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:36.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:36Z" } 2023-09-30T00:30:36.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:36.412 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:36.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:36.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:41.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:41.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:41.414 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:41.467 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:41.468 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:41.501 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:41.500+0000 7fccd572a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:41.502 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:41.501+0000 7fccd572a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:41.517 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:41.517 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:41.517 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:41.518 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.519 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:41.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:41.522 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:41.523 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:41.524 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:41.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:41.524 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:41Z" 2023-09-30T00:30:41.524 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:41.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:41Z" } 2023-09-30T00:30:41.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:41.525 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:41.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:41.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:46.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:46.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:46.527 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:46.580 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:46.581 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:46.614 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:46.613+0000 7ff1f5b33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:46.614 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:46.614+0000 7ff1f5b33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:46.630 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:46.631 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:46.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:46.633 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:46.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:46Z" 2023-09-30T00:30:46.636 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:46.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:46Z" } 2023-09-30T00:30:46.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:46.638 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:46.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:46.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:51.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:51.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:51.640 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:51.692 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:51.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:51.726 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:51.725+0000 7fba6ef21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:51.727 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:51.726+0000 7fba6ef21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:51.742 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:51.742 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:51.742 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:51.742 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:51.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:51.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:51.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:51.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:51.748 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:51Z" 2023-09-30T00:30:51.749 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:51.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:51Z" } 2023-09-30T00:30:51.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:51.751 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:51.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:51.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:30:56.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:56.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:30:56.753 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:30:56.806 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:30:56.806 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:30:56.838 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:56.838+0000 7f0de7412880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:56.839 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:30:56.839+0000 7f0de7412880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:30:56.854 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:30:56.854 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:30:56.854 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:30:56.854 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:56.855 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:30:56.857 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:56.858 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:56.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:30:56.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:56.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:56.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:30:56.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:30:56.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:30:56.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:30:56.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:30:56.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:30:56.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:30:56.862 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:30:56.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:30:56.862 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:30:56Z" 2023-09-30T00:30:56.862 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:30:56.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:30:56Z" } 2023-09-30T00:30:56.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:30:56.863 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:56.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:30:56.863 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:01.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:01.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:01.865 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:01.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:01.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:01.951 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:01.950+0000 7f1f44d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:01.952 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:01.951+0000 7f1f44d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:01.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:01Z" 2023-09-30T00:31:01.973 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:01.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:01Z" } 2023-09-30T00:31:01.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:01.975 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:01.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:01.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:06.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:06.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:06.976 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:07.029 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:07.029 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:07.063 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:07.062+0000 7fe2d8d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:07.064 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:07.063+0000 7fe2d8d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:07.078 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:07.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:07.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:07.081 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:07.082 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:07.083 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:07.084 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:07.085 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:07Z" 2023-09-30T00:31:07.085 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:07.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:07Z" } 2023-09-30T00:31:07.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:07.086 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:07.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:07.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:12.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:12.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:12.088 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:12.141 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:12.141 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:12.174 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:12.174+0000 7fbc6c824880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:12.175 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:12.175+0000 7fbc6c824880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:12.190 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:12Z" 2023-09-30T00:31:12.197 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:12.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:12Z" } 2023-09-30T00:31:12.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:12.198 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:12.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:12.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:17.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:17.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:17.200 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:17.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:17.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:17.287 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:17.286+0000 7f74aef22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:17.288 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:17.287+0000 7f74aef22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:17.303 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:17.304 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:17.305 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:17.306 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:17.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:17Z" 2023-09-30T00:31:17.309 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:17.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:17Z" } 2023-09-30T00:31:17.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:17.311 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:17.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:17.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:22.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:22.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:22.312 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:22.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:22.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:22.399 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:22.398+0000 7f7ade415880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:22.400 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:22.399+0000 7f7ade415880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:22.414 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:22.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:22.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:22.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:22.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:22.419 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:22.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:22.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:22.422 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:22Z" 2023-09-30T00:31:22.422 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:22.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:22Z" } 2023-09-30T00:31:22.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:22.423 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:22.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:22.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:27.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:27.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:27.425 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:27.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:27.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:27.511 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:27.511+0000 7fe4a0327880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:27.512 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:27.512+0000 7fe4a0327880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:27.527 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:27.527 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:27.527 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:27.527 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:27.527 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:27.528 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:27.529 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:27.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:27.531 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:27.532 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:27.533 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:27Z" 2023-09-30T00:31:27.534 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:27.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:27Z" } 2023-09-30T00:31:27.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:27.536 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:27.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:27.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:32.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:32.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:32.537 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:32.591 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:32.591 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:32.624 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:32.624+0000 7f647b527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:32.625 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:32.624+0000 7f647b527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:32.641 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:32.641 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:32.641 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:32Z" 2023-09-30T00:31:32.648 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:32.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:32Z" } 2023-09-30T00:31:32.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:32.650 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:32.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:32.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:37.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:37.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:37.651 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:37.705 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:37.705 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:37.737+0000 7fc0f3a15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:37.739 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:37.738+0000 7fc0f3a15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:37.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:37.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:37.756 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:37.757 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033684.864312_917.1", 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.864312Z", 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:37Z" 2023-09-30T00:31:37.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:37.761 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033684.864312_917.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.864312Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:37Z" } 2023-09-30T00:31:37.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033684.864312_917.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.864312Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:37.762 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:37.762 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033684.864312_917.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:37.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033810.358365_974.1 target=1_1696033684.864312_917.1 2023-09-30T00:31:42.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:42.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:42.763 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:42.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:42.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:42.850 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:42.850+0000 7fe215c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:42.851 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:42.851+0000 7fe215c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:42.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:42.866 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:42.866 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:42.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:42.867 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:42.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:42.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:42.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:42.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:42.872 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:42.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:42.873 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:42Z" 2023-09-30T00:31:42.873 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:42.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:42Z" } 2023-09-30T00:31:42.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:42.874 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:42.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:42.875 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:31:42.875 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:31:42.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:42.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:42.875 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:42.928 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:42.928 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:42.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:31:42.970+0000 7f70c929d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:42.971 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:31:42.971+0000 7f70c929d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:42.987 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:31:42.987 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:31:42.987 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:31:42.987 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:42.988 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:31:42.989 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:42.990 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:31:42.991 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:42.992 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:31:42.993 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:42.994 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:31:42Z" 2023-09-30T00:31:42.994 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:31:42.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:42Z" } 2023-09-30T00:31:42.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:42.995 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:42.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:42.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:31:42.996 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:42.996 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:42.996 DEBUG:boto:Method: HEAD 2023-09-30T00:31:42.996 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:42.996 DEBUG:boto:Data: 2023-09-30T00:31:42.996 DEBUG:boto:Headers: {} 2023-09-30T00:31:42.996 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:42.996 DEBUG:boto:Port: 8000 2023-09-30T00:31:42.997 DEBUG:boto:Params: {} 2023-09-30T00:31:42.997 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:31:42.997 DEBUG:boto:Token: None 2023-09-30T00:31:42.997 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:42 GMT /dmuhsu-81/ 2023-09-30T00:31:42.997 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5q4w1oWfP4VG3Iyij+NaTV0HCGE= 2023-09-30T00:31:42.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5q4w1oWfP4VG3Iyij+NaTV0HCGE=', 'Content-Length': '0'} 2023-09-30T00:31:43.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd9bc0c602addae9-0065176c6f-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.004 DEBUG:boto:path=/dmuhsu-81/obj7 2023-09-30T00:31:43.004 DEBUG:boto:auth_path=/dmuhsu-81/obj7 2023-09-30T00:31:43.004 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.004 DEBUG:boto:Path: /dmuhsu-81/obj7 2023-09-30T00:31:43.004 DEBUG:boto:Data: 2023-09-30T00:31:43.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.005 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:43.005 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.005 DEBUG:boto:Params: {} 2023-09-30T00:31:43.005 DEBUG:boto:Token: None 2023-09-30T00:31:43.005 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj7 2023-09-30T00:31:43.005 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cdVrx6yXTljWnX8Nuf05Eek6cwA= 2023-09-30T00:31:43.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cdVrx6yXTljWnX8Nuf05Eek6cwA='} 2023-09-30T00:31:43.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc049fa6770bcbcc-0065176c6f-2461-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.013 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:43.013 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:43.013 DEBUG:boto:Method: HEAD 2023-09-30T00:31:43.013 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:43.013 DEBUG:boto:Data: 2023-09-30T00:31:43.014 DEBUG:boto:Headers: {} 2023-09-30T00:31:43.014 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:43.014 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.014 DEBUG:boto:Params: {} 2023-09-30T00:31:43.014 DEBUG:boto:Token: None 2023-09-30T00:31:43.014 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/ 2023-09-30T00:31:43.014 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ= 2023-09-30T00:31:43.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ=', 'Content-Length': '0'} 2023-09-30T00:31:43.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b81a219a96d3bea-0065176c6f-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.017 DEBUG:boto:path=/dmuhsu-81/obj8 2023-09-30T00:31:43.017 DEBUG:boto:auth_path=/dmuhsu-81/obj8 2023-09-30T00:31:43.017 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.017 DEBUG:boto:Path: /dmuhsu-81/obj8 2023-09-30T00:31:43.017 DEBUG:boto:Data: 2023-09-30T00:31:43.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.017 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:43.017 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.017 DEBUG:boto:Params: {} 2023-09-30T00:31:43.018 DEBUG:boto:Token: None 2023-09-30T00:31:43.018 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj8 2023-09-30T00:31:43.018 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fSgpINvFfflDvHR48eRkRJlDaj0= 2023-09-30T00:31:43.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fSgpINvFfflDvHR48eRkRJlDaj0='} 2023-09-30T00:31:43.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000843f2c808e0af1d6-0065176c6f-2461-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.025 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:43.025 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:43.025 DEBUG:boto:Method: HEAD 2023-09-30T00:31:43.026 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:43.026 DEBUG:boto:Data: 2023-09-30T00:31:43.026 DEBUG:boto:Headers: {} 2023-09-30T00:31:43.026 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:43.026 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.026 DEBUG:boto:Params: {} 2023-09-30T00:31:43.026 DEBUG:boto:Token: None 2023-09-30T00:31:43.026 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/ 2023-09-30T00:31:43.026 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ= 2023-09-30T00:31:43.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ=', 'Content-Length': '0'} 2023-09-30T00:31:43.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aade051100c9704-0065176c6f-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.029 DEBUG:boto:path=/dmuhsu-81/obj9 2023-09-30T00:31:43.029 DEBUG:boto:auth_path=/dmuhsu-81/obj9 2023-09-30T00:31:43.029 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.029 DEBUG:boto:Path: /dmuhsu-81/obj9 2023-09-30T00:31:43.029 DEBUG:boto:Data: 2023-09-30T00:31:43.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.029 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:31:43.029 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.029 DEBUG:boto:Params: {} 2023-09-30T00:31:43.030 DEBUG:boto:Token: None 2023-09-30T00:31:43.030 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj9 2023-09-30T00:31:43.030 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D/MKBw216EzCfGuk+HAa7uPswUc= 2023-09-30T00:31:43.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D/MKBw216EzCfGuk+HAa7uPswUc='} 2023-09-30T00:31:43.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005225b9508cbd1da0-0065176c6f-2461-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.079 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:43.079 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:43.079 DEBUG:boto:Method: HEAD 2023-09-30T00:31:43.079 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:43.079 DEBUG:boto:Data: 2023-09-30T00:31:43.079 DEBUG:boto:Headers: {} 2023-09-30T00:31:43.079 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.079 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.080 DEBUG:boto:Params: {} 2023-09-30T00:31:43.080 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:31:43.080 DEBUG:boto:Token: None 2023-09-30T00:31:43.080 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/ 2023-09-30T00:31:43.081 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ= 2023-09-30T00:31:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ=', 'Content-Length': '0'} 2023-09-30T00:31:43.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ae88db6c7c00656-0065176c6f-1cc7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.086 DEBUG:boto:path=/dmuhsu-81/obj10 2023-09-30T00:31:43.087 DEBUG:boto:auth_path=/dmuhsu-81/obj10 2023-09-30T00:31:43.087 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.087 DEBUG:boto:Path: /dmuhsu-81/obj10 2023-09-30T00:31:43.087 DEBUG:boto:Data: 2023-09-30T00:31:43.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.087 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.087 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.087 DEBUG:boto:Params: {} 2023-09-30T00:31:43.087 DEBUG:boto:Token: None 2023-09-30T00:31:43.088 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj10 2023-09-30T00:31:43.088 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qb3NsqXvXcqNyql4SkbNCohZgNU= 2023-09-30T00:31:43.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qb3NsqXvXcqNyql4SkbNCohZgNU='} 2023-09-30T00:31:43.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3be0d8d95e30f24-0065176c6f-1cc7-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.095 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:43.095 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:43.095 DEBUG:boto:Method: HEAD 2023-09-30T00:31:43.095 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:43.095 DEBUG:boto:Data: 2023-09-30T00:31:43.095 DEBUG:boto:Headers: {} 2023-09-30T00:31:43.095 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.095 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.095 DEBUG:boto:Params: {} 2023-09-30T00:31:43.096 DEBUG:boto:Token: None 2023-09-30T00:31:43.096 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/ 2023-09-30T00:31:43.096 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ= 2023-09-30T00:31:43.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ=', 'Content-Length': '0'} 2023-09-30T00:31:43.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eab32a37868bc0e-0065176c6f-1cc7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.098 DEBUG:boto:path=/dmuhsu-81/obj11 2023-09-30T00:31:43.099 DEBUG:boto:auth_path=/dmuhsu-81/obj11 2023-09-30T00:31:43.099 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.099 DEBUG:boto:Path: /dmuhsu-81/obj11 2023-09-30T00:31:43.099 DEBUG:boto:Data: 2023-09-30T00:31:43.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.099 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.099 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.099 DEBUG:boto:Params: {} 2023-09-30T00:31:43.099 DEBUG:boto:Token: None 2023-09-30T00:31:43.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj11 2023-09-30T00:31:43.100 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:f4ocyZckukgGddtO3Hb3FpZvDk4= 2023-09-30T00:31:43.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:f4ocyZckukgGddtO3Hb3FpZvDk4='} 2023-09-30T00:31:43.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086e6b57d8d9204b1-0065176c6f-1cc7-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.106 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:31:43.106 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:31:43.106 DEBUG:boto:Method: HEAD 2023-09-30T00:31:43.106 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:31:43.106 DEBUG:boto:Data: 2023-09-30T00:31:43.107 DEBUG:boto:Headers: {} 2023-09-30T00:31:43.107 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.107 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.107 DEBUG:boto:Params: {} 2023-09-30T00:31:43.107 DEBUG:boto:Token: None 2023-09-30T00:31:43.107 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/ 2023-09-30T00:31:43.107 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ= 2023-09-30T00:31:43.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCqzMghrCae4yD5EFDU6L2eN8tQ=', 'Content-Length': '0'} 2023-09-30T00:31:43.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009762560c9477abef-0065176c6f-1cc7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.110 DEBUG:boto:path=/dmuhsu-81/obj12 2023-09-30T00:31:43.110 DEBUG:boto:auth_path=/dmuhsu-81/obj12 2023-09-30T00:31:43.110 DEBUG:boto:Method: PUT 2023-09-30T00:31:43.110 DEBUG:boto:Path: /dmuhsu-81/obj12 2023-09-30T00:31:43.110 DEBUG:boto:Data: 2023-09-30T00:31:43.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:31:43.110 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:31:43.110 DEBUG:boto:Port: 8000 2023-09-30T00:31:43.111 DEBUG:boto:Params: {} 2023-09-30T00:31:43.111 DEBUG:boto:Token: None 2023-09-30T00:31:43.111 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:31:43 GMT /dmuhsu-81/obj12 2023-09-30T00:31:43.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZHztHme72MSGOnwI9KCYq6yxsvE= 2023-09-30T00:31:43.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:31:43 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZHztHme72MSGOnwI9KCYq6yxsvE='} 2023-09-30T00:31:43.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005855641c3aeae6f1-0065176c6f-1cc7-test-zone2'), ('Date', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:31:43.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.159 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:43.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:43.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:43.268 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:31:43.285 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:31:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:43.285 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:30:10.358365Z" 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:43.287 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:31:43.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.287 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:31:43.288 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:43.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:31:43.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.288 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:43.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:43.336 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:43.366 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:43.365+0000 7fabb3213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:43.366 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:31:43.366+0000 7fabb3213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:43.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:43.380 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:31:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.385 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:43.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:43Z" 2023-09-30T00:31:43.387 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:43.388 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:43Z" } 2023-09-30T00:31:43.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:43.388 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:43.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:43.389 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:31:43.389 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:31:43.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.389 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:43.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:43.435 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:43.485 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:31:43.485+0000 7f8f2d867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:43.486 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:31:43.485+0000 7f8f2d867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:31:43.512 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.513 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:31:43.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:43.515 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:31:43.516 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:31:43.517 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.518 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:31:43.519 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:31:43.519 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:31:43.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.519 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:31:43Z" 2023-09-30T00:31:43.519 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:31:43.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:31:43Z" } 2023-09-30T00:31:43.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:31:43.520 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:43.520 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:31:43.521 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:31:43.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.521 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:43.566 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:43.566 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:43.647 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:31:43.649 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:31:43.650 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:43.650 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:31:43.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:43.651 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:43.700 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:43.700 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:31:43.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:43.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:43.749 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.750 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:43.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:31:43.751 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:31:43.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:43.751 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:43Z" 2023-09-30T00:31:43.751 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:43.752 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:31:43Z" } 2023-09-30T00:31:43.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:31:43.752 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:43.752 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:43.753 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:31:48.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:48.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:48.754 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:48.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:48.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:48.860 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:48.860 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.861 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:31:48.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:48.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:48.865 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:31:48.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.867 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:48Z" 2023-09-30T00:31:48.868 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:48.869 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:31:48Z" } 2023-09-30T00:31:48.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:31:48.870 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:48.870 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:48.870 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:31:53.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:53.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:53.871 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:53.926 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:53.926 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:53.975 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:53.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:53.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:53.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:31:53.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:53.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:31:53.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:53.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:53.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:53Z" 2023-09-30T00:31:53.998 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:53.999 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:31:53Z" } 2023-09-30T00:31:53.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:31:54.000 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:54.000 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:54.000 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:31:59.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:59.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:31:59.001 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:31:59.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:31:59.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:31:59.102 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:31:59.103 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:59.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:31:59.105 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:59.106 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:31:59.107 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:31:59Z" 2023-09-30T00:31:59.108 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:31:59.109 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:31:59Z" } 2023-09-30T00:31:59.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:31:59.110 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:59.110 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:31:59.110 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:32:04.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:04.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:04.111 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:04.165 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:04.165 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:04.215 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:04.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:04.217 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:32:04.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:04.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:04.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:04Z" 2023-09-30T00:32:04.222 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:04.223 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:04Z" } 2023-09-30T00:32:04.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:04.223 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:04.224 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:04.224 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:32:09.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:09.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:09.225 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:09.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:09.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:09.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013230", 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:09.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:27:39.718746Z" 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023527", 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:09.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.805633Z" 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021547", 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:09.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:16:08.594859Z" 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:09.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:09.336 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:09.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:09.337 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:09Z" 2023-09-30T00:32:09.337 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:09.338 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013230", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:27:39.718746Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023527", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:28:04.805633Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021547", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:16:08.594859Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:09Z" } 2023-09-30T00:32:09.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013230', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:27:39.718746Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023527', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.805633Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021547', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:16:08.594859Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:09.338 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:09.338 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013230', 1: '00000000000000000000:00000000000000023527', 2: '00000000000000000000:00000000000000021547', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:09.338 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013370 target=00000000000000000000:00000000000000013230, shard=1 master=00000000000000000000:00000000000000023667 target=00000000000000000000:00000000000000023527, shard=2 master=00000000000000000000:00000000000000021688 target=00000000000000000000:00000000000000021547 2023-09-30T00:32:14.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.340 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:14.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:14.396 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:14Z" 2023-09-30T00:32:14.452 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:14.453 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:14Z" } 2023-09-30T00:32:14.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:14.453 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:14.453 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:14.454 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:32:14.454 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:32:14.454 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:32:14.454 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.454 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:32:14.454 DEBUG:boto:Data: 2023-09-30T00:32:14.454 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.454 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.454 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.455 DEBUG:boto:Params: {} 2023-09-30T00:32:14.455 DEBUG:boto:Token: None 2023-09-30T00:32:14.455 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/ 2023-09-30T00:32:14.455 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y0Lp8K9b6stkgsEoSEiInAHmSYQ= 2023-09-30T00:32:14.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y0Lp8K9b6stkgsEoSEiInAHmSYQ=', 'Content-Length': '0'} 2023-09-30T00:32:14.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e82e4814cc4b2e1-0065176c8e-1cc7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.458 DEBUG:boto:path=/dmuhsu-81/obj7 2023-09-30T00:32:14.458 DEBUG:boto:auth_path=/dmuhsu-81/obj7 2023-09-30T00:32:14.458 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.458 DEBUG:boto:Path: /dmuhsu-81/obj7 2023-09-30T00:32:14.458 DEBUG:boto:Data: 2023-09-30T00:32:14.458 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.458 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.459 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.459 DEBUG:boto:Params: {} 2023-09-30T00:32:14.459 DEBUG:boto:Token: None 2023-09-30T00:32:14.459 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj7 2023-09-30T00:32:14.459 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:S6vL88mrsTeRkVnccH/Dpc7dtAw= 2023-09-30T00:32:14.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:S6vL88mrsTeRkVnccH/Dpc7dtAw=', 'Content-Length': '0'} 2023-09-30T00:32:14.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000730febf8685c1d99-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.461 DEBUG:boto:path=/dmuhsu-81/obj7 2023-09-30T00:32:14.461 DEBUG:boto:auth_path=/dmuhsu-81/obj7 2023-09-30T00:32:14.461 DEBUG:boto:Method: GET 2023-09-30T00:32:14.461 DEBUG:boto:Path: /dmuhsu-81/obj7 2023-09-30T00:32:14.462 DEBUG:boto:Data: 2023-09-30T00:32:14.462 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.462 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.462 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.462 DEBUG:boto:Params: {} 2023-09-30T00:32:14.462 DEBUG:boto:Token: None 2023-09-30T00:32:14.462 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj7 2023-09-30T00:32:14.462 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mQgX9uc7igc44FQB+Zla7bPNULA= 2023-09-30T00:32:14.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mQgX9uc7igc44FQB+Zla7bPNULA=', 'Content-Length': '0'} 2023-09-30T00:32:14.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002089d3396eedde25-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.465 DEBUG:boto:path=/dmuhsu-81/obj8 2023-09-30T00:32:14.465 DEBUG:boto:auth_path=/dmuhsu-81/obj8 2023-09-30T00:32:14.465 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.465 DEBUG:boto:Path: /dmuhsu-81/obj8 2023-09-30T00:32:14.465 DEBUG:boto:Data: 2023-09-30T00:32:14.465 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.465 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.465 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.465 DEBUG:boto:Params: {} 2023-09-30T00:32:14.466 DEBUG:boto:Token: None 2023-09-30T00:32:14.466 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj8 2023-09-30T00:32:14.466 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SzChWCQz4p7YfseUVjlu9ji1tOE= 2023-09-30T00:32:14.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SzChWCQz4p7YfseUVjlu9ji1tOE=', 'Content-Length': '0'} 2023-09-30T00:32:14.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031d6ece7635ba7d6-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.468 DEBUG:boto:path=/dmuhsu-81/obj8 2023-09-30T00:32:14.468 DEBUG:boto:auth_path=/dmuhsu-81/obj8 2023-09-30T00:32:14.468 DEBUG:boto:Method: GET 2023-09-30T00:32:14.468 DEBUG:boto:Path: /dmuhsu-81/obj8 2023-09-30T00:32:14.468 DEBUG:boto:Data: 2023-09-30T00:32:14.468 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.468 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.469 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.469 DEBUG:boto:Params: {} 2023-09-30T00:32:14.469 DEBUG:boto:Token: None 2023-09-30T00:32:14.469 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj8 2023-09-30T00:32:14.469 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8AT4gIdpcXwG6TTrPHAzgT7cI1M= 2023-09-30T00:32:14.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8AT4gIdpcXwG6TTrPHAzgT7cI1M=', 'Content-Length': '0'} 2023-09-30T00:32:14.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085f756a8cad99b29-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.471 DEBUG:boto:path=/dmuhsu-81/obj9 2023-09-30T00:32:14.471 DEBUG:boto:auth_path=/dmuhsu-81/obj9 2023-09-30T00:32:14.471 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.471 DEBUG:boto:Path: /dmuhsu-81/obj9 2023-09-30T00:32:14.471 DEBUG:boto:Data: 2023-09-30T00:32:14.471 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.472 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.472 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.472 DEBUG:boto:Params: {} 2023-09-30T00:32:14.472 DEBUG:boto:Token: None 2023-09-30T00:32:14.472 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj9 2023-09-30T00:32:14.472 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vMz50yf532afFDGljn5+a2GygsQ= 2023-09-30T00:32:14.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vMz50yf532afFDGljn5+a2GygsQ=', 'Content-Length': '0'} 2023-09-30T00:32:14.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007eb51b60cd5e5740-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.474 DEBUG:boto:path=/dmuhsu-81/obj9 2023-09-30T00:32:14.474 DEBUG:boto:auth_path=/dmuhsu-81/obj9 2023-09-30T00:32:14.474 DEBUG:boto:Method: GET 2023-09-30T00:32:14.474 DEBUG:boto:Path: /dmuhsu-81/obj9 2023-09-30T00:32:14.475 DEBUG:boto:Data: 2023-09-30T00:32:14.475 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.475 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:32:14.475 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.475 DEBUG:boto:Params: {} 2023-09-30T00:32:14.475 DEBUG:boto:Token: None 2023-09-30T00:32:14.475 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj9 2023-09-30T00:32:14.475 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Gr28IkbqFIBtmuHjuu0F/uiWrsA= 2023-09-30T00:32:14.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Gr28IkbqFIBtmuHjuu0F/uiWrsA=', 'Content-Length': '0'} 2023-09-30T00:32:14.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-81:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9203.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000249c050febb10dd3-0065176c8e-1cc7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.477 DEBUG:boto:path=/dmuhsu-81/ 2023-09-30T00:32:14.477 DEBUG:boto:auth_path=/dmuhsu-81/ 2023-09-30T00:32:14.477 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.478 DEBUG:boto:Path: /dmuhsu-81/ 2023-09-30T00:32:14.478 DEBUG:boto:Data: 2023-09-30T00:32:14.478 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.478 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:14.478 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.478 DEBUG:boto:Params: {} 2023-09-30T00:32:14.478 DEBUG:boto:Token: None 2023-09-30T00:32:14.478 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/ 2023-09-30T00:32:14.479 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:y0Lp8K9b6stkgsEoSEiInAHmSYQ= 2023-09-30T00:32:14.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y0Lp8K9b6stkgsEoSEiInAHmSYQ=', 'Content-Length': '0'} 2023-09-30T00:32:14.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042a337a1c8a70421-0065176c8e-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.481 DEBUG:boto:path=/dmuhsu-81/obj10 2023-09-30T00:32:14.482 DEBUG:boto:auth_path=/dmuhsu-81/obj10 2023-09-30T00:32:14.482 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.482 DEBUG:boto:Path: /dmuhsu-81/obj10 2023-09-30T00:32:14.482 DEBUG:boto:Data: 2023-09-30T00:32:14.482 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.482 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:14.482 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.483 DEBUG:boto:Params: {} 2023-09-30T00:32:14.483 DEBUG:boto:Token: None 2023-09-30T00:32:14.483 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj10 2023-09-30T00:32:14.483 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nZPa/jElubdpWJurpcn5+680YLU= 2023-09-30T00:32:14.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nZPa/jElubdpWJurpcn5+680YLU=', 'Content-Length': '0'} 2023-09-30T00:32:14.485 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005b1cc5a28f74ac86-0065176c8e-2461-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.485 DEBUG:boto:path=/dmuhsu-81/obj11 2023-09-30T00:32:14.485 DEBUG:boto:auth_path=/dmuhsu-81/obj11 2023-09-30T00:32:14.485 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.485 DEBUG:boto:Path: /dmuhsu-81/obj11 2023-09-30T00:32:14.485 DEBUG:boto:Data: 2023-09-30T00:32:14.486 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.486 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:14.486 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.486 DEBUG:boto:Params: {} 2023-09-30T00:32:14.486 DEBUG:boto:Token: None 2023-09-30T00:32:14.486 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj11 2023-09-30T00:32:14.486 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:D5q11SYAPjq+SiwMuiPHOSkIYuc= 2023-09-30T00:32:14.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:D5q11SYAPjq+SiwMuiPHOSkIYuc=', 'Content-Length': '0'} 2023-09-30T00:32:14.488 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000001c97ee0f82bd1df5-0065176c8e-2461-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.488 DEBUG:boto:path=/dmuhsu-81/obj12 2023-09-30T00:32:14.488 DEBUG:boto:auth_path=/dmuhsu-81/obj12 2023-09-30T00:32:14.488 DEBUG:boto:Method: HEAD 2023-09-30T00:32:14.488 DEBUG:boto:Path: /dmuhsu-81/obj12 2023-09-30T00:32:14.488 DEBUG:boto:Data: 2023-09-30T00:32:14.489 DEBUG:boto:Headers: {} 2023-09-30T00:32:14.489 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:14.489 DEBUG:boto:Port: 8000 2023-09-30T00:32:14.489 DEBUG:boto:Params: {} 2023-09-30T00:32:14.489 DEBUG:boto:Token: None 2023-09-30T00:32:14.489 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:14 GMT /dmuhsu-81/obj12 2023-09-30T00:32:14.489 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:xIu7Qcgzf2AlxsWUvobquhFnBP8= 2023-09-30T00:32:14.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:14 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xIu7Qcgzf2AlxsWUvobquhFnBP8=', 'Content-Length': '0'} 2023-09-30T00:32:14.491 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000db2e07e0e959db59-0065176c8e-2461-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:32:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:14.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:14.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:14.492 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dmuhsu-81 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:14.544 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:14.545 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:14.599 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:32:14.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:14.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:14.601 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:14.645 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:14.645 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:14.704 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:32:14.705 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... ok 2023-09-30T00:32:14.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.707 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:14.789 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:14.789 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:14.838 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:32:14.858 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:32:14.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:14.858 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033934.583688_978.1", 2023-09-30T00:32:14.859 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:14.583688Z" 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:14.860 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:32:14.861 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:14.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:32:14.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:14.862 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:14.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:14.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:14.946 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:14.945+0000 7efe35723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:14.947 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:14.946+0000 7efe35723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:14.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:32:14.962 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:14.963 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:14.964 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:14.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:32:14.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:14.968 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:14Z" 2023-09-30T00:32:14.968 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:14.969 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:14Z" } 2023-09-30T00:32:14.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:14.969 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:14.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:14.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033934.583688_978.1 target=1_1696033810.358365_974.1 2023-09-30T00:32:19.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:19.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:19.971 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:20.023 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:20.056 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:20.055+0000 7f759a726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:20.057 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:20.056+0000 7f759a726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:20.072 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:20.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:20.074 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:20.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033810.358365_974.1", 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:30:10.358365Z", 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:20.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:20Z" 2023-09-30T00:32:20.078 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:20.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033810.358365_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:30:10.358365Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:20Z" } 2023-09-30T00:32:20.080 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033810.358365_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:30:10.358365Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:20.080 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:20.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033810.358365_974.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:20.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696033934.583688_978.1 target=1_1696033810.358365_974.1 2023-09-30T00:32:25.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.081 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:25.137 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.137 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.170 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:25.169+0000 7f4973867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.170 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:25.170+0000 7f4973867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.185 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:25.185 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:25.185 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:25.185 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033934.583688_978.1", 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:14.583688Z", 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.190 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:25.191 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.192 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:25.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:32:25.193 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:32:25.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.193 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:25Z" 2023-09-30T00:32:25.193 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:25.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033934.583688_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:14.583688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:25Z" } 2023-09-30T00:32:25.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033934.583688_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:14.583688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:25.195 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.195 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:32:25.195 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:32:25.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.195 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:25.252 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.252 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.286 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:25.285+0000 7fcaf4f2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.287 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:25.286+0000 7fcaf4f2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.302 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.303 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:25.304 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.305 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033934.583688_978.1", 2023-09-30T00:32:25.306 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:14.583688Z", 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.307 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.308 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:32:25.309 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:32:25.309 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:32:25.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.309 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:32:25Z" 2023-09-30T00:32:25.309 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:32:25.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033934.583688_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:14.583688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:25Z" } 2023-09-30T00:32:25.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033934.583688_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:14.583688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:25.311 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.311 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:32:25.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:25.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:25.312 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:25.355 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.355 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.429 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:32:25.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:25.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:25.430 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:25.511 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.512 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.576 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:32:25.576 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-83 2023-09-30T00:32:25.576 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:32:25.577 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:32:25.577 DEBUG:boto:Method: PUT 2023-09-30T00:32:25.577 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:32:25.577 DEBUG:boto:Data: 2023-09-30T00:32:25.577 DEBUG:boto:Headers: {} 2023-09-30T00:32:25.577 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:25.577 DEBUG:boto:Port: 8000 2023-09-30T00:32:25.577 DEBUG:boto:Params: {} 2023-09-30T00:32:25.578 DEBUG:boto:Token: None 2023-09-30T00:32:25.578 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:32:25 GMT /dmuhsu-83/ 2023-09-30T00:32:25.578 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:dzzcGImtlac6JzM1tZWzU5QCz28= 2023-09-30T00:32:25.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dzzcGImtlac6JzM1tZWzU5QCz28=', 'Content-Length': '0'} 2023-09-30T00:32:25.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087658b32199728e0-0065176c99-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:25.601 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-84 2023-09-30T00:32:25.601 DEBUG:boto:path=/dmuhsu-84/ 2023-09-30T00:32:25.601 DEBUG:boto:auth_path=/dmuhsu-84/ 2023-09-30T00:32:25.601 DEBUG:boto:Method: PUT 2023-09-30T00:32:25.601 DEBUG:boto:Path: /dmuhsu-84/ 2023-09-30T00:32:25.601 DEBUG:boto:Data: 2023-09-30T00:32:25.601 DEBUG:boto:Headers: {} 2023-09-30T00:32:25.602 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:25.602 DEBUG:boto:Port: 8000 2023-09-30T00:32:25.602 DEBUG:boto:Params: {} 2023-09-30T00:32:25.602 DEBUG:boto:Token: None 2023-09-30T00:32:25.602 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:32:25 GMT /dmuhsu-84/ 2023-09-30T00:32:25.602 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:EBTcBAUkwL1ePan/+AQbL4OHh2c= 2023-09-30T00:32:25.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EBTcBAUkwL1ePan/+AQbL4OHh2c=', 'Content-Length': '0'} 2023-09-30T00:32:25.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e1fb991d123b787-0065176c99-2461-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:25.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.619 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.663 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.709 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:32:25.728 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:32:25.728 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.728 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:25.728 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:25.616943Z" 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.729 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:32:25.730 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.730 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:32:25.731 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:32:25.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.732 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:25.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.823 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:25.823+0000 7f3f3f410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.824 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:25.824+0000 7f3f3f410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.838 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:25.838 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.839 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.840 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:25.841 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.842 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:25.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:25.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:25Z" 2023-09-30T00:32:25.845 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:25.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:25Z" } 2023-09-30T00:32:25.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:25.851 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.851 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:32:25.851 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:32:25.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:25.852 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:25.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:25.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:25.950 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:25.949+0000 7f3b41d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.951 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:25.950+0000 7f3b41d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:32:25.970 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:32:25.970 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:32:25.970 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:32:25.970 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:32:25.970 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:25.971 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.971 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:32:25.971 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.972 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:32:25.972 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.972 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.972 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:25.972 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.973 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:32:25.973 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.973 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:25.973 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.973 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:25.974 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.975 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033934.583688_978.1", 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:14.583688Z", 2023-09-30T00:32:25.976 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:25.977 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.978 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:32:25.979 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:32:25.979 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:32:25.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:25.979 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:32:25Z" 2023-09-30T00:32:25.979 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:32:25.980 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033934.583688_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:14.583688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:25Z" } 2023-09-30T00:32:25.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033934.583688_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:14.583688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:25.981 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.981 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033934.583688_978.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:25.981 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696033945.616943_986.1 target=1_1696033934.583688_978.1 2023-09-30T00:32:30.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:30.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:30.982 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:31.035 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.035 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.081 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:31.080+0000 7feb49213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:31.082 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:31.081+0000 7feb49213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:31.102 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:32:31.102 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:32:31.102 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:32:31.102 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:31.103 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:32:31.104 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.105 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:32:31.106 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:31.107 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:32:31Z" 2023-09-30T00:32:31.108 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:32:31.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:31Z" } 2023-09-30T00:32:31.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:31.110 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:31.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:31.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:32:31.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.110 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.156 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.156 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.217 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:32:31.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.217 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.268 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.268 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.337 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:32:31.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=dmuhsu-83', '--dest-bucket=dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=dmuhsu-83', '--dest-bucket=dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.337 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --source-bucket=dmuhsu-83 --dest-bucket=dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.387 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.388 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.447 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:32:31.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.448 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.494 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.551 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:32:31.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.551 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.593 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.594 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.656 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-83","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:32:31.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.657 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.741 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.741 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.791 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:32:31.791 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:31.791 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 15, 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:32:31.792 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.793 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:32:31.794 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:32:31.796 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.797 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:32:31.798 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.799 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.800 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.801 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:32:31.802 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:32:31.802 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.807 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:32:31.808 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-83", 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-83", 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:32:31.809 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.810 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.811 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:32:31.812 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:31.813 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:32:31.814 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:32:31.815 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:32:31.815 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:32:31.815 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:32:31.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.815 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:32:31.816 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:32:31.817 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:31.818 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:32:31.819 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:32:31.819 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:31.819 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:32:31.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:32:31.820 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:32:31.861 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:31.861 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:31.943 INFO:teuthology.orchestra.run.smithi175.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"test-zone2","bucket":"dmuhsu-83"},"dest":{"zone":"test-zone1","bucket":"dmuhsu-83"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"test-zone1","bucket":"dmuhsu-83"},"dest":{"zone":"test-zone2","bucket":"dmuhsu-83"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-09-30T00:32:31.944 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:32:31.944 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:32:31.944 DEBUG:boto:Method: HEAD 2023-09-30T00:32:31.944 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:32:31.944 DEBUG:boto:Data: 2023-09-30T00:32:31.944 DEBUG:boto:Headers: {} 2023-09-30T00:32:31.944 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:31.944 DEBUG:boto:Port: 8000 2023-09-30T00:32:31.945 DEBUG:boto:Params: {} 2023-09-30T00:32:31.945 DEBUG:boto:Token: None 2023-09-30T00:32:31.945 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:31 GMT /dmuhsu-83/ 2023-09-30T00:32:31.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:4G0PRRUFZiul1s7wcMWDVRLkQqI= 2023-09-30T00:32:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4G0PRRUFZiul1s7wcMWDVRLkQqI=', 'Content-Length': '0'} 2023-09-30T00:32:32.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24c5b36fcc33d28-0065176ca0-253f-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.164 DEBUG:boto:path=/dmuhsu-83/obj1 2023-09-30T00:32:32.164 DEBUG:boto:auth_path=/dmuhsu-83/obj1 2023-09-30T00:32:32.164 DEBUG:boto:Method: PUT 2023-09-30T00:32:32.165 DEBUG:boto:Path: /dmuhsu-83/obj1 2023-09-30T00:32:32.165 DEBUG:boto:Data: 2023-09-30T00:32:32.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:32:32.165 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.165 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.165 DEBUG:boto:Params: {} 2023-09-30T00:32:32.165 DEBUG:boto:Token: None 2023-09-30T00:32:32.165 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-83/obj1 2023-09-30T00:32:32.166 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Qfo2+PYpVorVqgv0QcDLjHbE+DI= 2023-09-30T00:32:32.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Qfo2+PYpVorVqgv0QcDLjHbE+DI='} 2023-09-30T00:32:32.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff9414a1a760b51f-0065176ca0-253f-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.205 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:32:32.206 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:32:32.206 DEBUG:boto:Method: HEAD 2023-09-30T00:32:32.206 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:32:32.206 DEBUG:boto:Data: 2023-09-30T00:32:32.206 DEBUG:boto:Headers: {} 2023-09-30T00:32:32.206 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.206 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.206 DEBUG:boto:Params: {} 2023-09-30T00:32:32.207 DEBUG:boto:Token: None 2023-09-30T00:32:32.207 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-83/ 2023-09-30T00:32:32.207 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s0gqy3vH2r8oTmXYbo1FVGijGlc= 2023-09-30T00:32:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s0gqy3vH2r8oTmXYbo1FVGijGlc=', 'Content-Length': '0'} 2023-09-30T00:32:32.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ddeaaaab3b34721-0065176ca0-253f-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.211 DEBUG:boto:path=/dmuhsu-83/obj2 2023-09-30T00:32:32.211 DEBUG:boto:auth_path=/dmuhsu-83/obj2 2023-09-30T00:32:32.212 DEBUG:boto:Method: PUT 2023-09-30T00:32:32.212 DEBUG:boto:Path: /dmuhsu-83/obj2 2023-09-30T00:32:32.212 DEBUG:boto:Data: 2023-09-30T00:32:32.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:32:32.212 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.212 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.212 DEBUG:boto:Params: {} 2023-09-30T00:32:32.212 DEBUG:boto:Token: None 2023-09-30T00:32:32.213 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-83/obj2 2023-09-30T00:32:32.213 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QeTNixNjLT8cS8gNopMKdrJHiBw= 2023-09-30T00:32:32.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QeTNixNjLT8cS8gNopMKdrJHiBw='} 2023-09-30T00:32:32.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a3613a19fc19de-0065176ca0-253f-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.259 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:32:32.259 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:32:32.259 DEBUG:boto:Method: HEAD 2023-09-30T00:32:32.259 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:32:32.259 DEBUG:boto:Data: 2023-09-30T00:32:32.259 DEBUG:boto:Headers: {} 2023-09-30T00:32:32.259 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.259 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.260 DEBUG:boto:Params: {} 2023-09-30T00:32:32.260 DEBUG:boto:Token: None 2023-09-30T00:32:32.260 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-83/ 2023-09-30T00:32:32.260 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:s0gqy3vH2r8oTmXYbo1FVGijGlc= 2023-09-30T00:32:32.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s0gqy3vH2r8oTmXYbo1FVGijGlc=', 'Content-Length': '0'} 2023-09-30T00:32:32.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007381fd80e84e0903-0065176ca0-253f-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.264 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:32:32.264 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:32:32.264 DEBUG:boto:Method: PUT 2023-09-30T00:32:32.264 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:32:32.264 DEBUG:boto:Data: 2023-09-30T00:32:32.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:32:32.265 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.265 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.265 DEBUG:boto:Params: {} 2023-09-30T00:32:32.265 DEBUG:boto:Token: None 2023-09-30T00:32:32.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-83/obj3 2023-09-30T00:32:32.265 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zCOVqqMXmIbvqCD8rF5zezmSExk= 2023-09-30T00:32:32.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zCOVqqMXmIbvqCD8rF5zezmSExk='} 2023-09-30T00:32:32.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd1ca7a7d4174453-0065176ca0-253f-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.285 DEBUG:boto:path=/dmuhsu-84/ 2023-09-30T00:32:32.285 DEBUG:boto:auth_path=/dmuhsu-84/ 2023-09-30T00:32:32.285 DEBUG:boto:Method: HEAD 2023-09-30T00:32:32.285 DEBUG:boto:Path: /dmuhsu-84/ 2023-09-30T00:32:32.285 DEBUG:boto:Data: 2023-09-30T00:32:32.285 DEBUG:boto:Headers: {} 2023-09-30T00:32:32.285 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.285 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.286 DEBUG:boto:Params: {} 2023-09-30T00:32:32.286 DEBUG:boto:Token: None 2023-09-30T00:32:32.286 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-84/ 2023-09-30T00:32:32.286 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:cebyifPy4M0iW8oyh62gAUo7Qt8= 2023-09-30T00:32:32.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cebyifPy4M0iW8oyh62gAUo7Qt8=', 'Content-Length': '0'} 2023-09-30T00:32:32.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004251bc057ed874b6-0065176ca0-253f-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.295 DEBUG:boto:path=/dmuhsu-84/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-30T00:32:32.295 DEBUG:boto:auth_path=/dmuhsu-84/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-30T00:32:32.295 DEBUG:boto:Method: PUT 2023-09-30T00:32:32.295 DEBUG:boto:Path: /dmuhsu-84/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-30T00:32:32.295 DEBUG:boto:Data: 2023-09-30T00:32:32.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:32:32.295 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:32:32.295 DEBUG:boto:Port: 8000 2023-09-30T00:32:32.295 DEBUG:boto:Params: {} 2023-09-30T00:32:32.296 DEBUG:boto:Token: None 2023-09-30T00:32:32.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:32:32 GMT /dmuhsu-84/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-30T00:32:32.296 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:WsEXP1f0qWeDkz+lc/GNyScRBFI= 2023-09-30T00:32:32.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:32:32 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WsEXP1f0qWeDkz+lc/GNyScRBFI='} 2023-09-30T00:32:32.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052aaa395b70a66c0-0065176ca0-253f-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:32:32.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.339 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:32.410 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:32.410 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:32.509 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:32:32.549 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.550 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:25.616943Z" 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.551 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:32:32.552 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:32.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:32:32.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.553 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:32.598 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:32.598 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:32.628 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:32.627+0000 7f2538e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:32.629 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:32:32.628+0000 7f2538e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:32.642 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:32.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:32.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:32.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:32.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:32.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:32.648 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:32Z" 2023-09-30T00:32:32.649 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:32.650 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:32Z" } 2023-09-30T00:32:32.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:32.650 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:32.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:32.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:32:32.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:32:32.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.651 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:32.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:32.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:32.776 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:32.774+0000 7fbc93a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:32.777 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:32:32.776+0000 7fbc93a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:32:32.798 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:32:32.798 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:32:32.798 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:32:32.798 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:32.799 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.800 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:32.801 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:32.802 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.803 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.804 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.804 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:32:32.804 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:32:32.804 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:32:32.805 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:32:32.806 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.808 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:32:32Z" 2023-09-30T00:32:32.808 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:32:32.809 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:32:32Z" } 2023-09-30T00:32:32.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:32:32.810 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:32.810 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:32:32.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:32:32.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.811 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:32.859 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:32.860 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:32.977 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:32:32.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.977 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:32.977 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000023947", 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.279871Z" 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.978 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:32:32.979 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:32:32.981 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:32.981 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:32:32.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:32.981 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:33.030 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:33.031 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:33.076 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:33.076 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:33.076 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:33.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:33.078 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:33.078 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:33.078 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:33.078 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:33.079 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:33.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:33.079 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:33.079 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:33.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:33.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:33.080 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:33.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:33.080 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:33.081 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:33.081 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:33.081 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:33.081 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:33.082 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:33.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:33.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:33.083 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:33.083 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:33.083 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:33.083 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:33.084 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:33.085 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:33.085 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:33.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:33.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:33.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:33.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:33.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:33.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:33.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:33.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:33.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:33.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:33Z" 2023-09-30T00:32:33.089 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:33.091 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:33Z" } 2023-09-30T00:32:33.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:33.092 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:33.092 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:33.092 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:32:38.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:38.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:38.094 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:38.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:38.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:38.199 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:38.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:38.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:38.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:38.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:38.204 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:38.205 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:38Z" 2023-09-30T00:32:38.206 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:38.207 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:38Z" } 2023-09-30T00:32:38.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:38.207 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:38.207 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:38.207 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:32:43.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:43.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:43.209 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:43.265 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:43.265 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:43.314 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:43.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:43.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:43.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:43Z" 2023-09-30T00:32:43.321 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:43.322 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:43Z" } 2023-09-30T00:32:43.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:43.322 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:43.322 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:43.322 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:32:48.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:48.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:48.324 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:48.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:48.379 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:48.429 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:48.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:48.429 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:48.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:48.429 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:48.430 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:48.431 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:48.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:48Z" 2023-09-30T00:32:48.435 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:48.436 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:48Z" } 2023-09-30T00:32:48.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:48.436 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:48.436 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:48.437 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:32:53.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:53.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:53.438 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:53.493 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:53.493 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.545 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:53.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:53.547 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:53.548 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:53.549 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:53.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:53Z" 2023-09-30T00:32:53.551 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:53.552 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:53Z" } 2023-09-30T00:32:53.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:53.552 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:53.552 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:53.553 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:32:58.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:58.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:32:58.554 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:32:58.610 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:32:58.610 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:32:58.660 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:32:58.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:32:58.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:32:58.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:32:58.665 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:32:58.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:32:58.667 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:32:58Z" 2023-09-30T00:32:58.667 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:32:58.668 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:32:58Z" } 2023-09-30T00:32:58.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:32:58.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:32:58.669 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:32:58.669 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:03.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:03.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:03.670 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.776 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:03.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:03.778 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:03.779 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:03.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:03Z" 2023-09-30T00:33:03.782 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:03.783 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:03Z" } 2023-09-30T00:33:03.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:03.783 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:03.783 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:03.784 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:08.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:08.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:08.785 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:08.840 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:08.841 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:08.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:08.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:08.893 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:08.894 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:08.895 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:08.896 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:08.897 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:08Z" 2023-09-30T00:33:08.897 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:08.898 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:08Z" } 2023-09-30T00:33:08.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:08.898 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:08.898 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:08.898 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:13.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:13.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:13.900 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:13.954 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:13.955 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:14.005 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:13Z" 2023-09-30T00:33:14.012 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:14.013 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:13Z" } 2023-09-30T00:33:14.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:14.014 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:14.014 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:14.014 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:19.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:19.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:19.015 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:19.070 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:19.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:19.120 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:19.120 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:19.120 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:19.121 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:19.122 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:19.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:19.125 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:19Z" 2023-09-30T00:33:19.126 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:19.127 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:19Z" } 2023-09-30T00:33:19.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:19.127 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:19.128 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:19.128 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:24.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:24.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:24.129 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:24.186 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:24.186 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.237 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:24.238 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:24.239 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:24.240 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:24.241 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:24.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:24.243 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:24Z" 2023-09-30T00:33:24.243 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:24.244 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:24Z" } 2023-09-30T00:33:24.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:24.244 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:24.244 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:24.244 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:29.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:29.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:29.245 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:29.300 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:29.300 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:29.350 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:29.350 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:29.351 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:29.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:29.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:29.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:29Z" 2023-09-30T00:33:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:29.357 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:29Z" } 2023-09-30T00:33:29.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:29.358 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:29.358 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:29.358 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:34.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:34.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:34.360 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:34.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:34.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:34.465 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:34.465 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:34.465 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:34.465 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:34.466 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:34.467 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:34.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:34.471 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:34Z" 2023-09-30T00:33:34.472 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:34.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:34Z" } 2023-09-30T00:33:34.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:34.473 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:34.473 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:34.473 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:39.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:39.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:39.474 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:39.530 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:39.531 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:39.581 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:39.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:39.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:39.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:39.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:39.586 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:39.587 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:39Z" 2023-09-30T00:33:39.588 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:39.589 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:39Z" } 2023-09-30T00:33:39.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:39.590 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:39.590 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:39.590 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:44.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:44.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:44.591 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:44.646 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:44.647 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:44.697 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:44.698 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:44.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:44.700 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:44.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:44.702 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:44.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:44.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:44.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:44.703 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:44.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:44Z" 2023-09-30T00:33:44.704 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:44.705 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:44Z" } 2023-09-30T00:33:44.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:44.706 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:44.706 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:44.706 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:49.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:49.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:49.707 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:49.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:49.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:49.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:49.812 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:49.812 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:49.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:49.812 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:49.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:49.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:49.816 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:49.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:49.818 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:49.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:49.819 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:49Z" 2023-09-30T00:33:49.819 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:49.820 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:49Z" } 2023-09-30T00:33:49.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:49.820 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:49.820 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:49.820 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:54.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:54.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:54.822 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:54.879 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:54.879 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:33:54.929 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:33:54.929 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:54.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:54.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:54.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:54.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:33:54.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:33:54.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:33:54Z" 2023-09-30T00:33:54.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:33:54.937 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:33:54Z" } 2023-09-30T00:33:54.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:33:54.937 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:33:54.938 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:33:54.938 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:33:59.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:59.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:33:59.939 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:33:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:33:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:00.045 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:00.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:00.047 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:00.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:00.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:00Z" 2023-09-30T00:34:00.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:00.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:00Z" } 2023-09-30T00:34:00.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:00.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:00.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:00.053 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:05.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:05.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:05.054 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:05.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:05.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:05.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:05.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:05.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:05.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:05.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:05.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:05.170 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:05Z" 2023-09-30T00:34:05.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:05.171 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:05Z" } 2023-09-30T00:34:05.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:05.171 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:05.171 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:05.171 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:10.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:10.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:10.173 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:10.228 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:10.228 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:10.279 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:10.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:10.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:10.284 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:10Z" 2023-09-30T00:34:10.285 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:10.286 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:10Z" } 2023-09-30T00:34:10.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:10.286 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:10.287 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:10.287 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:15.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:15.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:15.288 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:15.344 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:15.344 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:15.458 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:15.459 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:15.460 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:15.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:15.462 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:15.463 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:15.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:15.465 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:15Z" 2023-09-30T00:34:15.465 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:15.466 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:15Z" } 2023-09-30T00:34:15.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:15.466 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:15.466 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:15.466 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:20.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:20.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:20.468 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:20.523 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:20.523 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:20.577 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:20.577 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:20.577 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:20.577 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:20.578 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:20.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:20.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:20.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:20.582 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:20.583 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:20.584 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:20Z" 2023-09-30T00:34:20.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:20.585 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:20Z" } 2023-09-30T00:34:20.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:20.585 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:20.585 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:20.586 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:25.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:25.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:25.587 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:25.642 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:25.642 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:25.691 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:25.691 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:25.691 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:25.692 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:25.693 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:25.694 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:25.695 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:25.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:25Z" 2023-09-30T00:34:25.697 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:25.698 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:25Z" } 2023-09-30T00:34:25.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:25.699 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:25.699 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:25.699 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:30.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:30.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:30.701 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:30.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:30.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:30.805 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023667", 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.012677Z" 2023-09-30T00:34:30.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021688", 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:30.809 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.036121Z" 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018194", 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:30.810 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:28:04.757082Z" 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:30Z" 2023-09-30T00:34:30.811 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:30.812 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023667", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:31:43.012677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021688", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:31:43.036121Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:28:04.757082Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:30Z" } 2023-09-30T00:34:30.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023667', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.012677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021688', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.036121Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:28:04.757082Z'}}] 2023-09-30T00:34:30.813 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:30.813 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023667', 2: '00000000000000000000:00000000000000021688', 3: '00000000000000000000:00000000000000018194'} 2023-09-30T00:34:30.813 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000023947 target=00000000000000000000:00000000000000023667, shard=2 master=00000000000000000000:00000000000000021828 target=00000000000000000000:00000000000000021688, shard=3 master=00000000000000000000:00000000000000018334 target=00000000000000000000:00000000000000018194 2023-09-30T00:34:35.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:35.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:35.814 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:35.869 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:35.869 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:35.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:35.919 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:35.919 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:35.920 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:35.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000023947", 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.279871Z" 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:35.922 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:35.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:35.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:35Z" 2023-09-30T00:34:35.925 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:35.927 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023947", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:32:32.279871Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:34:35Z" } 2023-09-30T00:34:35.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023947', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.279871Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:34:35.927 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:35.928 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000023947', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:34:35.928 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:34:35.928 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:34:35.928 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:34:35.928 DEBUG:boto:Method: HEAD 2023-09-30T00:34:35.928 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:34:35.928 DEBUG:boto:Data: 2023-09-30T00:34:35.929 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.929 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.929 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.929 DEBUG:boto:Params: {} 2023-09-30T00:34:35.929 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:34:35.929 DEBUG:boto:Token: None 2023-09-30T00:34:35.930 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/ 2023-09-30T00:34:35.930 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:djsP4pxZk47bKL7y5WEJmLafW3M= 2023-09-30T00:34:35.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:djsP4pxZk47bKL7y5WEJmLafW3M=', 'Content-Length': '0'} 2023-09-30T00:34:35.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093675825fdfd4fb9-0065176d1b-1dc2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.936 DEBUG:boto:path=/dmuhsu-83/obj1 2023-09-30T00:34:35.936 DEBUG:boto:auth_path=/dmuhsu-83/obj1 2023-09-30T00:34:35.936 DEBUG:boto:Method: HEAD 2023-09-30T00:34:35.937 DEBUG:boto:Path: /dmuhsu-83/obj1 2023-09-30T00:34:35.937 DEBUG:boto:Data: 2023-09-30T00:34:35.937 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.937 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.937 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.937 DEBUG:boto:Params: {} 2023-09-30T00:34:35.937 DEBUG:boto:Token: None 2023-09-30T00:34:35.937 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj1 2023-09-30T00:34:35.938 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:od4NVofUiw6O/WGoYOmTyIxCRR0= 2023-09-30T00:34:35.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:od4NVofUiw6O/WGoYOmTyIxCRR0=', 'Content-Length': '0'} 2023-09-30T00:34:35.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f4b756228e1b7b9-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.940 DEBUG:boto:path=/dmuhsu-83/obj1 2023-09-30T00:34:35.940 DEBUG:boto:auth_path=/dmuhsu-83/obj1 2023-09-30T00:34:35.940 DEBUG:boto:Method: GET 2023-09-30T00:34:35.940 DEBUG:boto:Path: /dmuhsu-83/obj1 2023-09-30T00:34:35.940 DEBUG:boto:Data: 2023-09-30T00:34:35.940 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.941 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.941 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.941 DEBUG:boto:Params: {} 2023-09-30T00:34:35.941 DEBUG:boto:Token: None 2023-09-30T00:34:35.941 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj1 2023-09-30T00:34:35.941 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+xsea/7hueQr4Idead5XTpOAKUc= 2023-09-30T00:34:35.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+xsea/7hueQr4Idead5XTpOAKUc=', 'Content-Length': '0'} 2023-09-30T00:34:35.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de5933516b0efe9e-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.943 DEBUG:boto:path=/dmuhsu-83/obj2 2023-09-30T00:34:35.943 DEBUG:boto:auth_path=/dmuhsu-83/obj2 2023-09-30T00:34:35.943 DEBUG:boto:Method: HEAD 2023-09-30T00:34:35.944 DEBUG:boto:Path: /dmuhsu-83/obj2 2023-09-30T00:34:35.944 DEBUG:boto:Data: 2023-09-30T00:34:35.944 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.944 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.944 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.944 DEBUG:boto:Params: {} 2023-09-30T00:34:35.944 DEBUG:boto:Token: None 2023-09-30T00:34:35.944 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj2 2023-09-30T00:34:35.945 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8JEORmQr7Y+Eptc/sLkf1Ou5N70= 2023-09-30T00:34:35.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8JEORmQr7Y+Eptc/sLkf1Ou5N70=', 'Content-Length': '0'} 2023-09-30T00:34:35.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccecffe2837641a9-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.946 DEBUG:boto:path=/dmuhsu-83/obj2 2023-09-30T00:34:35.946 DEBUG:boto:auth_path=/dmuhsu-83/obj2 2023-09-30T00:34:35.947 DEBUG:boto:Method: GET 2023-09-30T00:34:35.947 DEBUG:boto:Path: /dmuhsu-83/obj2 2023-09-30T00:34:35.947 DEBUG:boto:Data: 2023-09-30T00:34:35.947 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.947 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.948 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.948 DEBUG:boto:Params: {} 2023-09-30T00:34:35.949 DEBUG:boto:Token: None 2023-09-30T00:34:35.949 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj2 2023-09-30T00:34:35.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:jwHICJTDL+Tdbs8ueoYQGb9DkyA= 2023-09-30T00:34:35.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jwHICJTDL+Tdbs8ueoYQGb9DkyA=', 'Content-Length': '0'} 2023-09-30T00:34:35.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004305424890718dcc-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.951 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:34:35.951 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:34:35.951 DEBUG:boto:Method: HEAD 2023-09-30T00:34:35.951 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:34:35.951 DEBUG:boto:Data: 2023-09-30T00:34:35.951 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.951 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.952 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.952 DEBUG:boto:Params: {} 2023-09-30T00:34:35.952 DEBUG:boto:Token: None 2023-09-30T00:34:35.952 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj3 2023-09-30T00:34:35.952 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:G421627eXl58eVRaAs0v4rHofqw= 2023-09-30T00:34:35.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:G421627eXl58eVRaAs0v4rHofqw=', 'Content-Length': '0'} 2023-09-30T00:34:35.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000420b322cd1ea90b3-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.955 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:34:35.955 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:34:35.955 DEBUG:boto:Method: GET 2023-09-30T00:34:35.955 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:34:35.955 DEBUG:boto:Data: 2023-09-30T00:34:35.955 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.955 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.955 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.955 DEBUG:boto:Params: {} 2023-09-30T00:34:35.956 DEBUG:boto:Token: None 2023-09-30T00:34:35.956 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:34:35 GMT /dmuhsu-83/obj3 2023-09-30T00:34:35.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1rgCGPCTVT+RJ6WPuDxgL3iHobc= 2023-09-30T00:34:35.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1rgCGPCTVT+RJ6WPuDxgL3iHobc=', 'Content-Length': '0'} 2023-09-30T00:34:35.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:32:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cacfe86c840c1562-0065176d1b-1dc2-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:35.998 DEBUG:boto:path=/dmuhsu-84/ 2023-09-30T00:34:35.998 DEBUG:boto:auth_path=/dmuhsu-84/ 2023-09-30T00:34:35.999 DEBUG:boto:Method: HEAD 2023-09-30T00:34:35.999 DEBUG:boto:Path: /dmuhsu-84/ 2023-09-30T00:34:35.999 DEBUG:boto:Data: 2023-09-30T00:34:35.999 DEBUG:boto:Headers: {} 2023-09-30T00:34:35.999 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:35.999 DEBUG:boto:Port: 8000 2023-09-30T00:34:35.999 DEBUG:boto:Params: {} 2023-09-30T00:34:35.999 DEBUG:boto:Token: None 2023-09-30T00:34:36.000 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:36 GMT /dmuhsu-84/ 2023-09-30T00:34:36.000 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/k0It1IcBV77Zsoa3r7MfXuP2Hc= 2023-09-30T00:34:36.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/k0It1IcBV77Zsoa3r7MfXuP2Hc=', 'Content-Length': '0'} 2023-09-30T00:34:36.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd8990a7aecc982b-0065176d1c-1dc2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:36.003 DEBUG:boto:path=/dmuhsu-84/obj1 2023-09-30T00:34:36.003 DEBUG:boto:auth_path=/dmuhsu-84/obj1 2023-09-30T00:34:36.003 DEBUG:boto:Method: HEAD 2023-09-30T00:34:36.003 DEBUG:boto:Path: /dmuhsu-84/obj1 2023-09-30T00:34:36.003 DEBUG:boto:Data: 2023-09-30T00:34:36.003 DEBUG:boto:Headers: {} 2023-09-30T00:34:36.003 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:36.003 DEBUG:boto:Port: 8000 2023-09-30T00:34:36.004 DEBUG:boto:Params: {} 2023-09-30T00:34:36.004 DEBUG:boto:Token: None 2023-09-30T00:34:36.004 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:36 GMT /dmuhsu-84/obj1 2023-09-30T00:34:36.004 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nH6xmgof8x1Dq87Uq/jPc1p2CzQ= 2023-09-30T00:34:36.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nH6xmgof8x1Dq87Uq/jPc1p2CzQ=', 'Content-Length': '0'} 2023-09-30T00:34:36.005 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000175f0815e29bf1fb-0065176d1c-1dc2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:36.005 DEBUG:boto:path=/dmuhsu-84/obj2 2023-09-30T00:34:36.006 DEBUG:boto:auth_path=/dmuhsu-84/obj2 2023-09-30T00:34:36.006 DEBUG:boto:Method: HEAD 2023-09-30T00:34:36.006 DEBUG:boto:Path: /dmuhsu-84/obj2 2023-09-30T00:34:36.006 DEBUG:boto:Data: 2023-09-30T00:34:36.006 DEBUG:boto:Headers: {} 2023-09-30T00:34:36.007 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:36.007 DEBUG:boto:Port: 8000 2023-09-30T00:34:36.007 DEBUG:boto:Params: {} 2023-09-30T00:34:36.007 DEBUG:boto:Token: None 2023-09-30T00:34:36.007 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:36 GMT /dmuhsu-84/obj2 2023-09-30T00:34:36.007 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:bT7Ctq67rZUxuL44Rb6xIeLKMvY= 2023-09-30T00:34:36.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bT7Ctq67rZUxuL44Rb6xIeLKMvY=', 'Content-Length': '0'} 2023-09-30T00:34:36.008 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000050caf1e03470e437-0065176d1c-1dc2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:36.009 DEBUG:boto:path=/dmuhsu-84/obj3 2023-09-30T00:34:36.009 DEBUG:boto:auth_path=/dmuhsu-84/obj3 2023-09-30T00:34:36.009 DEBUG:boto:Method: HEAD 2023-09-30T00:34:36.009 DEBUG:boto:Path: /dmuhsu-84/obj3 2023-09-30T00:34:36.009 DEBUG:boto:Data: 2023-09-30T00:34:36.009 DEBUG:boto:Headers: {} 2023-09-30T00:34:36.009 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:34:36.009 DEBUG:boto:Port: 8000 2023-09-30T00:34:36.009 DEBUG:boto:Params: {} 2023-09-30T00:34:36.010 DEBUG:boto:Token: None 2023-09-30T00:34:36.010 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:36 GMT /dmuhsu-84/obj3 2023-09-30T00:34:36.010 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:W7f+vo+afDEeNCJaVCyAiA5Qn6w= 2023-09-30T00:34:36.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7f+vo+afDEeNCJaVCyAiA5Qn6w=', 'Content-Length': '0'} 2023-09-30T00:34:36.011 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c85d97b2cfb18f1b-0065176d1c-1dc2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:36.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.012 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.069 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.070 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.135 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"enabled"}]} 2023-09-30T00:34:36.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.136 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.184 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.185 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.253 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:34:36.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.254 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.299 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.299 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.371 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:34:36.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.372 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.415 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.416 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.505 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 16, 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:34:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:34:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:34:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:34:36.511 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.512 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:34:36.513 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.514 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:34:36.515 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:34:36.516 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:34:36.517 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:34:36.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.532 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:34:36.533 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "allowed" 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.535 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:34:36.536 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:34:36.537 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:34:36.538 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:34:36.539 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:34:36.540 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:34:36.541 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:34:36.542 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:34:36.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.543 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.589 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.589 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.689 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:34:36.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.690 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:36.749 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:36.749 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:36.930 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:34:36.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:36.931 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:37.006 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.006 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.156 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:34:37.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.157 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:37.216 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.323 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:34:37.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.323 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:37.380 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.380 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.504 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:34:37.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:34:37.505 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:34:37.554 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.554 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.661 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"*","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:34:37.661 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:34:37.661 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:34:37.661 DEBUG:boto:Method: HEAD 2023-09-30T00:34:37.661 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:34:37.661 DEBUG:boto:Data: 2023-09-30T00:34:37.662 DEBUG:boto:Headers: {} 2023-09-30T00:34:37.662 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:34:37.662 DEBUG:boto:Port: 8000 2023-09-30T00:34:37.662 DEBUG:boto:Params: {} 2023-09-30T00:34:37.662 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:34:37.662 DEBUG:boto:Token: None 2023-09-30T00:34:37.662 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:37 GMT /dmuhsu-83/ 2023-09-30T00:34:37.663 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:99Dc4ufx3RihKCrGAtlTjaWXRvw= 2023-09-30T00:34:37.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:99Dc4ufx3RihKCrGAtlTjaWXRvw=', 'Content-Length': '0'} 2023-09-30T00:34:37.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002299a976a223883-0065176d1d-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:37.668 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:34:37.668 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:34:37.668 DEBUG:boto:Method: PUT 2023-09-30T00:34:37.668 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:34:37.668 DEBUG:boto:Data: 2023-09-30T00:34:37.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:34:37.668 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:34:37.668 DEBUG:boto:Port: 8000 2023-09-30T00:34:37.669 DEBUG:boto:Params: {} 2023-09-30T00:34:37.669 DEBUG:boto:Token: None 2023-09-30T00:34:37.669 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:34:37 GMT /dmuhsu-83/obj3 2023-09-30T00:34:37.669 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wGGqJtIa7nIgk8U14WyKB2Tx9Tg= 2023-09-30T00:34:37.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:34:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wGGqJtIa7nIgk8U14WyKB2Tx9Tg='} 2023-09-30T00:34:37.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000635b3e7069550a87-0065176d1d-2585-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:37.695 DEBUG:boto:path=/dmuhsu-84/ 2023-09-30T00:34:37.695 DEBUG:boto:auth_path=/dmuhsu-84/ 2023-09-30T00:34:37.695 DEBUG:boto:Method: HEAD 2023-09-30T00:34:37.695 DEBUG:boto:Path: /dmuhsu-84/ 2023-09-30T00:34:37.695 DEBUG:boto:Data: 2023-09-30T00:34:37.695 DEBUG:boto:Headers: {} 2023-09-30T00:34:37.695 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:34:37.695 DEBUG:boto:Port: 8000 2023-09-30T00:34:37.695 DEBUG:boto:Params: {} 2023-09-30T00:34:37.696 DEBUG:boto:Token: None 2023-09-30T00:34:37.696 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:34:37 GMT /dmuhsu-84/ 2023-09-30T00:34:37.696 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CJLtQ+ulnujHHVuCFp9jaQE8TBU= 2023-09-30T00:34:37.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:34:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CJLtQ+ulnujHHVuCFp9jaQE8TBU=', 'Content-Length': '0'} 2023-09-30T00:34:37.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095ebf36b13c68340-0065176d1d-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:37.699 DEBUG:boto:path=/dmuhsu-84/obj3 2023-09-30T00:34:37.699 DEBUG:boto:auth_path=/dmuhsu-84/obj3 2023-09-30T00:34:37.699 DEBUG:boto:Method: PUT 2023-09-30T00:34:37.699 DEBUG:boto:Path: /dmuhsu-84/obj3 2023-09-30T00:34:37.700 DEBUG:boto:Data: 2023-09-30T00:34:37.700 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:34:37.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:34:37.700 DEBUG:boto:Port: 8000 2023-09-30T00:34:37.700 DEBUG:boto:Params: {} 2023-09-30T00:34:37.700 DEBUG:boto:Token: None 2023-09-30T00:34:37.700 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:34:37 GMT /dmuhsu-84/obj3 2023-09-30T00:34:37.700 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1SVqeuyc+jzk3+HTXWOJw1g0XJA= 2023-09-30T00:34:37.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:34:37 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1SVqeuyc+jzk3+HTXWOJw1g0XJA='} 2023-09-30T00:34:37.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080e3ac7b43151c4d-0065176d1d-2585-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:34:37.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:37.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:37.743 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:37.787 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.788 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.822 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:34:37.835 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:34:37.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:37.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:34:37.470578Z" 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:34:37.837 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:34:37.838 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:37.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:34:37.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:37.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:37.839 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:37.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:37.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:37.918 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:37.917+0000 7fb75e217880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:37.918 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:37.918+0000 7fb75e217880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:37.932 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:37.932 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:37.932 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:37.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:37.932 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:37.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:37.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:37.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:37.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:34:37.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:37.939 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:37Z" 2023-09-30T00:34:37.939 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:37.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:34:37Z" } 2023-09-30T00:34:37.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:34:37.940 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:37.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:37.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:34:42.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:42.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:42.942 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:42.995 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:42.996 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:43.029 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:43.028+0000 7f9d1371e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:43.030 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:43.029+0000 7f9d1371e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:43.044 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:43Z" 2023-09-30T00:34:43.051 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:43.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:34:43Z" } 2023-09-30T00:34:43.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:34:43.054 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:43.054 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:43.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:34:48.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:48.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:48.056 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:48.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:48.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:48.142 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:48.141+0000 7f52a6f33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:48.143 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:48.142+0000 7f52a6f33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:48.157 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:48.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:48.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:48.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:48.161 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:48.162 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:48.163 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:48Z" 2023-09-30T00:34:48.164 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:48.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:34:48Z" } 2023-09-30T00:34:48.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:34:48.165 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:48.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:48.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:34:53.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:53.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:53.167 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:53.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:53.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:53.256 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:53.255+0000 7f28eac2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:53.256 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:53.256+0000 7f28eac2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:53.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:53.271 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:53.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:53.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:53.275 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:53.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:34:53.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:53.278 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:53Z" 2023-09-30T00:34:53.278 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:53.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:34:53Z" } 2023-09-30T00:34:53.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:34:53.279 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:53.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:53.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:34:58.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:58.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:34:58.282 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:34:58.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:34:58.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:34:58.371 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:58.370+0000 7f9123c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:58.371 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:34:58.371+0000 7f9123c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:34:58.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:34:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:34:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:34:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:34:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:34:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:34:58Z" 2023-09-30T00:34:58.395 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:34:58.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:34:58Z" } 2023-09-30T00:34:58.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:34:58.396 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:58.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:34:58.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:03.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:03.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:03.398 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:03.452 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:03.485 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:03.485+0000 7fa463926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:03.486 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:03.486+0000 7fa463926880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:03.500 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:03.501 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:03.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:03.503 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:03.504 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:03.505 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:03.506 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:03.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:03.507 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:03Z" 2023-09-30T00:35:03.507 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:03.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:03Z" } 2023-09-30T00:35:03.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:03.508 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:03.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:03.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:08.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:08.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:08.510 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:08.564 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:08.578 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:08.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:08.597+0000 7f04ef32b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:08.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:08.598+0000 7f04ef32b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:08.613 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:08.613 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:08.613 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:08.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:08.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:08.616 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:08.617 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:08.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:08.619 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:08.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:08.621 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:08Z" 2023-09-30T00:35:08.621 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:08.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:08Z" } 2023-09-30T00:35:08.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:08.622 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:08.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:08.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:13.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:13.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:13.624 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:13.676 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:13.676 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:13.710 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:13.709+0000 7f66efc15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:13.711 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:13.710+0000 7f66efc15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:13.725 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:13.725 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.726 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:13.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:13.728 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:13.729 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:13.730 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:13.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:13.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:13.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.734 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:13.735 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:13Z" 2023-09-30T00:35:13.735 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:13.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:13Z" } 2023-09-30T00:35:13.737 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:13.737 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:13.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:13.737 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:18.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:18.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:18.739 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:18.791 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:18.791 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:18.825 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:18.824+0000 7f27c6e67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:18.825+0000 7f27c6e67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:18.840 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:18.840 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:18.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:18.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:18.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:18.846 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:18.847 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:18Z" 2023-09-30T00:35:18.847 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:18.847 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:18Z" } 2023-09-30T00:35:18.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:18.848 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:18.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:18.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:23.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:23.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:23.849 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:23.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:23.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:23.935 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:23.934+0000 7fc651d11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:23.936 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:23.935+0000 7fc651d11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:23.950 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:23.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:23.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:23.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:23.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:23.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.956 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:23.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:23.957 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:23.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:23.957 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:23Z" 2023-09-30T00:35:23.957 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:23.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:23Z" } 2023-09-30T00:35:23.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:23.958 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:23.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:23.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:28.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:28.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:28.960 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:29.012 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:29.012 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:29.046 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:29.045+0000 7fb28420f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:29.046 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:29.046+0000 7fb28420f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:29.061 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:29.061 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:29.061 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:29.061 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:29.062 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:29.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:29.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:29.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:29.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:29.067 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:29.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:29.068 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:29Z" 2023-09-30T00:35:29.068 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:29.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:29Z" } 2023-09-30T00:35:29.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:29.069 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:29.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:29.069 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:34.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:34.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:34.070 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:34.123 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:34.123 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:34.157 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:34.156+0000 7f946ad26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:34.158 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:34.157+0000 7f946ad26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:34.172 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:34Z" 2023-09-30T00:35:34.179 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:34.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:34Z" } 2023-09-30T00:35:34.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:34.180 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:34.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:34.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:39.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:39.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:39.182 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:39.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:39.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:39.269 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:39.268+0000 7ff7df48b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:39.285 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:39.269+0000 7ff7df48b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:39.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:39.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:39Z" 2023-09-30T00:35:39.292 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:39.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:39Z" } 2023-09-30T00:35:39.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:39.293 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:39.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:39.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:44.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:44.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:44.295 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:44.348 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:44.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:44.382 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:44.381+0000 7f7bc7736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:44.382 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:44.382+0000 7f7bc7736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:44.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:44.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:44.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:44.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:44.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:44.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:44Z" 2023-09-30T00:35:44.403 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:44.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:44Z" } 2023-09-30T00:35:44.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:44.405 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:44.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:44.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:49.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:49.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:49.406 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:49.459 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:49.460 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:49.493 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:49.492+0000 7f8597215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:49.494 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:49.493+0000 7f8597215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:49.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:49.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:49.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:49.512 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:49.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:49.514 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:49Z" 2023-09-30T00:35:49.515 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:49.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:49Z" } 2023-09-30T00:35:49.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:49.517 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:49.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:49.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:54.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:54.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:54.519 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:54.572 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:54.572 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:54.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:54.605+0000 7fd982924880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:54.606 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:54.606+0000 7fd982924880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:54.621 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:54.622 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:54.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:54.624 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:54.625 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:54.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:54Z" 2023-09-30T00:35:54.627 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:54.628 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:54Z" } 2023-09-30T00:35:54.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:54.629 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:54.629 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:54.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:35:59.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:59.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:35:59.630 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:35:59.682 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:35:59.682 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:35:59.716 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:59.716+0000 7f8384c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:59.717 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:35:59.717+0000 7f8384c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:35:59.732 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:35:59.732 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:35:59.732 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:35:59.732 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:35:59.732 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:59.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:59.735 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:59.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:35:59.737 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.738 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:35:59.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:35:59.739 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:35:59.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:35:59.739 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:35:59Z" 2023-09-30T00:35:59.739 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:35:59.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:35:59Z" } 2023-09-30T00:35:59.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:35:59.740 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:59.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:35:59.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:04.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:04.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:04.742 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:04.794 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:04.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:04.828 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:04.828+0000 7ff3b8f26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:04.829 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:04.829+0000 7ff3b8f26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:04.844 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:04.844 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:04.844 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:04.844 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:04.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:04.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:04.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:04.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:04.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:04.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:04.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:04.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:04.850 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:04.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:04Z" 2023-09-30T00:36:04.852 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:04.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:04Z" } 2023-09-30T00:36:04.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:04.854 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:04.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:04.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:09.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:09.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:09.856 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:09.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:09.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:09.943 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:09.942+0000 7f1472622880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:09.944 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:09.943+0000 7f1472622880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:09.958 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:09.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:09.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:09.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:09.962 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:09.963 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:36:09.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:09.965 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:09Z" 2023-09-30T00:36:09.965 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:09.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:09Z" } 2023-09-30T00:36:09.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:09.966 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:09.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:09.966 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:36:09.966 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:36:09.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:09.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:09.967 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:10.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:10.019 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:10.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:10.056+0000 7efd19817880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:10.057 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:10.057+0000 7efd19817880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:10.077 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:10.078 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:10.079 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:10.080 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:10.081 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:10.082 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:10Z" 2023-09-30T00:36:10.083 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:10.084 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:10Z" } 2023-09-30T00:36:10.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:10.084 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:10.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:10.085 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:15.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:15.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:15.086 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:15.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:15.139 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:15.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:15.193+0000 7f4701927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:15.195 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:15.194+0000 7f4701927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:15.220 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:15.221 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:15.222 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:15.223 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:15.224 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:15Z" 2023-09-30T00:36:15.225 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:15.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:15Z" } 2023-09-30T00:36:15.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:15.226 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:15.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:15.227 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:20.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:20.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:20.228 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:20.284 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:20.284 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:20.324 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:20.322+0000 7f3827813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:20.325 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:20.323+0000 7f3827813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:20.340 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:20.340 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:20.340 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:20.340 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:20.340 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:20.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:20.342 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:20.343 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:20.344 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:20.345 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:20.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:20Z" 2023-09-30T00:36:20.347 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:20.348 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:20Z" } 2023-09-30T00:36:20.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:20.348 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:20.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:20.349 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:25.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:25.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:25.350 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:25.403 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:25.403 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:25.443 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:25.442+0000 7ff7350c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:25.444 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:25.443+0000 7ff7350c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:25.459 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:25.459 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.460 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:25.461 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:25.466 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:25.466 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:25.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:25.466 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:25Z" 2023-09-30T00:36:25.466 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:25.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:25Z" } 2023-09-30T00:36:25.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:25.467 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:25.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:25.468 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:30.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:30.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:30.469 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:30.522 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:30.522 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:30.567 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:30.566+0000 7fb00d12a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:30.568 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:30.566+0000 7fb00d12a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:30.585 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:30.585 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:30.586 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:30.588 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:30.589 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:30.590 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:30.591 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:30.592 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:30Z" 2023-09-30T00:36:30.592 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:30.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:30Z" } 2023-09-30T00:36:30.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:30.593 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:30.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:30.593 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:35.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:35.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:35.594 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:35.648 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:35.648 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:35.698 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:35.698+0000 7f8db5f2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:35.699 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:35.699+0000 7f8db5f2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:35.719 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:35.720 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:35.721 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:35.722 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696033945.616943_986.1", 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:32:25.616943Z", 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.723 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:35.724 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:35Z" 2023-09-30T00:36:35.725 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:35.726 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696033945.616943_986.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:25.616943Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:35Z" } 2023-09-30T00:36:35.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696033945.616943_986.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:25.616943Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:35.727 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:35.727 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696033945.616943_986.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:35.727 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034077.470578_1054.1 target=1_1696033945.616943_986.1 2023-09-30T00:36:40.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.728 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:40.780 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:40.781 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:40.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:40.828+0000 7f9641d21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:40.829 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:40.829+0000 7f9641d21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:40.847 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.848 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:40.849 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:40.850 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:40.851 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:40.852 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.853 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:40.854 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:40.854 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:40.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.854 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:40Z" 2023-09-30T00:36:40.854 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:40.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:40Z" } 2023-09-30T00:36:40.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:40.856 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:40.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:40.856 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:36:40.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.856 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:40.901 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:40.902 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:40.976 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:36:40.977 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:40.978 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:36:40.979 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:36:40.980 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:36:40.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:40.980 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:41.034 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:41.035 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:41.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:41.085 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:41.085 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:41.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:41.086 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:41.086 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:36:41.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.086 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:36:41.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:36:41.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:36:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:36:41.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:41.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:36:41.093 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:36:41.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.093 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:41Z" 2023-09-30T00:36:41.093 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:41.094 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:36:41Z" } 2023-09-30T00:36:41.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:36:41.094 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:36:41.094 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:36:41.095 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:36:41.095 DEBUG:boto:path=/dmuhsu-83/ 2023-09-30T00:36:41.095 DEBUG:boto:auth_path=/dmuhsu-83/ 2023-09-30T00:36:41.095 DEBUG:boto:Method: HEAD 2023-09-30T00:36:41.095 DEBUG:boto:Path: /dmuhsu-83/ 2023-09-30T00:36:41.095 DEBUG:boto:Data: 2023-09-30T00:36:41.095 DEBUG:boto:Headers: {} 2023-09-30T00:36:41.095 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:36:41.096 DEBUG:boto:Port: 8000 2023-09-30T00:36:41.096 DEBUG:boto:Params: {} 2023-09-30T00:36:41.096 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:36:41.096 DEBUG:boto:Token: None 2023-09-30T00:36:41.096 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:36:41 GMT /dmuhsu-83/ 2023-09-30T00:36:41.097 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:P+Fmy/9SX8FoUt1A+Akbh8WUCt0= 2023-09-30T00:36:41.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:36:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:P+Fmy/9SX8FoUt1A+Akbh8WUCt0=', 'Content-Length': '0'} 2023-09-30T00:36:41.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c419b31dd61938a9-0065176d99-1e96-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:36:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:36:41.102 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:36:41.103 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:36:41.103 DEBUG:boto:Method: HEAD 2023-09-30T00:36:41.103 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:36:41.103 DEBUG:boto:Data: 2023-09-30T00:36:41.103 DEBUG:boto:Headers: {} 2023-09-30T00:36:41.103 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:36:41.103 DEBUG:boto:Port: 8000 2023-09-30T00:36:41.103 DEBUG:boto:Params: {} 2023-09-30T00:36:41.104 DEBUG:boto:Token: None 2023-09-30T00:36:41.104 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:36:41 GMT /dmuhsu-83/obj3 2023-09-30T00:36:41.104 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:UpTd/8ew9bPT5rExbLDgKikgCTE= 2023-09-30T00:36:41.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:36:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UpTd/8ew9bPT5rExbLDgKikgCTE=', 'Content-Length': '0'} 2023-09-30T00:36:41.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086a17edd0e9c1885-0065176d99-1e96-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:36:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:36:41.106 DEBUG:boto:path=/dmuhsu-83/obj3 2023-09-30T00:36:41.106 DEBUG:boto:auth_path=/dmuhsu-83/obj3 2023-09-30T00:36:41.107 DEBUG:boto:Method: GET 2023-09-30T00:36:41.107 DEBUG:boto:Path: /dmuhsu-83/obj3 2023-09-30T00:36:41.107 DEBUG:boto:Data: 2023-09-30T00:36:41.107 DEBUG:boto:Headers: {} 2023-09-30T00:36:41.107 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:36:41.107 DEBUG:boto:Port: 8000 2023-09-30T00:36:41.107 DEBUG:boto:Params: {} 2023-09-30T00:36:41.107 DEBUG:boto:Token: None 2023-09-30T00:36:41.107 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:36:41 GMT /dmuhsu-83/obj3 2023-09-30T00:36:41.108 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wkbua4+SPLkyTzsXzwPXoSImHzo= 2023-09-30T00:36:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:36:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wkbua4+SPLkyTzsXzwPXoSImHzo=', 'Content-Length': '0'} 2023-09-30T00:36:41.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:34:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-83:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9317.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f38697d244a7aec-0065176d99-1e96-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:36:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:36:41.109 DEBUG:boto:path=/dmuhsu-84/ 2023-09-30T00:36:41.110 DEBUG:boto:auth_path=/dmuhsu-84/ 2023-09-30T00:36:41.110 DEBUG:boto:Method: HEAD 2023-09-30T00:36:41.110 DEBUG:boto:Path: /dmuhsu-84/ 2023-09-30T00:36:41.110 DEBUG:boto:Data: 2023-09-30T00:36:41.110 DEBUG:boto:Headers: {} 2023-09-30T00:36:41.110 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:36:41.110 DEBUG:boto:Port: 8000 2023-09-30T00:36:41.111 DEBUG:boto:Params: {} 2023-09-30T00:36:41.111 DEBUG:boto:Token: None 2023-09-30T00:36:41.111 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:36:41 GMT /dmuhsu-84/ 2023-09-30T00:36:41.111 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Ee4VvoRX+dxfCNl2C3nNB1HwVOI= 2023-09-30T00:36:41.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:36:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ee4VvoRX+dxfCNl2C3nNB1HwVOI=', 'Content-Length': '0'} 2023-09-30T00:36:41.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b23de3eb31ae5e3-0065176d99-1e96-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:36:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:36:41.113 DEBUG:boto:path=/dmuhsu-84/obj3 2023-09-30T00:36:41.114 DEBUG:boto:auth_path=/dmuhsu-84/obj3 2023-09-30T00:36:41.114 DEBUG:boto:Method: HEAD 2023-09-30T00:36:41.114 DEBUG:boto:Path: /dmuhsu-84/obj3 2023-09-30T00:36:41.114 DEBUG:boto:Data: 2023-09-30T00:36:41.114 DEBUG:boto:Headers: {} 2023-09-30T00:36:41.114 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:36:41.114 DEBUG:boto:Port: 8000 2023-09-30T00:36:41.114 DEBUG:boto:Params: {} 2023-09-30T00:36:41.115 DEBUG:boto:Token: None 2023-09-30T00:36:41.115 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:36:41 GMT /dmuhsu-84/obj3 2023-09-30T00:36:41.115 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a0PSz/Le+nARqx/VnBWSLIde5Oc= 2023-09-30T00:36:41.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:36:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a0PSz/Le+nARqx/VnBWSLIde5Oc=', 'Content-Length': '0'} 2023-09-30T00:36:41.116 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000094f6546e9c6b57c7-0065176d99-1e96-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:36:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:36:41.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:36:41.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:36:41.117 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dmuhsu-83 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:36:41.163 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:41.163 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:41.256 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:36:41.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:36:41.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:36:41.257 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:36:41.301 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:41.301 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:41.353 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:36:41.354 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-09-30T00:36:41.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:41.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:41.356 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:41.400 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:41.400 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:41.459 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:36:41.483 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:36:41.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:41.483 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:41.483 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:36:41.484 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:36:41.231011Z" 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:41.485 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:36:41.487 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:41.487 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:36:41.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:41.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:41.488 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:41.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:41.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:41.567+0000 7f37c751c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:41.568+0000 7f37c751c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:41.581 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:41.581 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:41.581 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:41.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:41.582 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:41.582 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:41.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:41.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.583 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:41.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:41.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.584 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:41.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:41.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:36:41.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:41.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:41.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:41.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.586 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:41.586 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:41.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:41.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:41.587 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:41.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:41.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.589 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:41.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:41.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:41.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:41.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:41.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:41.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:41.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:41.591 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:41.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:41.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:41Z" 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:41.596 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:41Z" } 2023-09-30T00:36:41.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:41.597 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:41.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:41.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:36:46.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:46.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:46.600 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:46.653 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:46.653 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:46.687 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:46.686+0000 7f53af723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:46.688 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:46.687+0000 7f53af723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:46.706 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:46.706 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:46.706 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:46.706 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:46.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:46.708 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:46.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:46.710 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:46.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:36:46.712 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:36:46.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:46.713 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:46Z" 2023-09-30T00:36:46.713 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:46.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:46Z" } 2023-09-30T00:36:46.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:46.713 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:46.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:46.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:36:51.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:51.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:51.715 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:51.768 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:51.769 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:51.802 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:51.801+0000 7fd199f27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:51.803 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:36:51.802+0000 7fd199f27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:36:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:36:51.819 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:51.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:51.821 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:36:51.822 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:51.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:36:51.824 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:36:51.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034201.231011_1059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:36:41.231011Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:51Z" } 2023-09-30T00:36:51.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034201.231011_1059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:36:41.231011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:51.825 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:51.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:51.826 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:36:51.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:36:51.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:51.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:51.826 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:51.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:51.879 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:51.916 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:51.916+0000 7f197d733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:51.917 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:51.917+0000 7f197d733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:51.932 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:51.932 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:51.933 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:51.934 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:51.936 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:51.937 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:36:51.938 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:51.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:51Z" } 2023-09-30T00:36:51.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:51.940 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:51.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:51.940 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:36:56.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:56.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:36:56.942 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:36:56.995 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:36:56.995 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:36:57.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:57.055+0000 7f1a5fe67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:57.056 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:36:57.056+0000 7f1a5fe67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:36:57.083 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.084 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:36:57.085 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:57.086 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:36:57.087 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.088 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:36:57.089 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:36:57Z" 2023-09-30T00:36:57.090 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:36:57.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034077.470578_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:34:37.470578Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:36:57Z" } 2023-09-30T00:36:57.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:36:57.092 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:57.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:36:57.092 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:37:02.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.094 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:02.149 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:02.149 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:02.186 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:02.185+0000 7f13a5467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:02.187 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:02.187+0000 7f13a5467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:37:02.213 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:02.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:02.216 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:02.217 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:02.218 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.219 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:37:02.220 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:37:02.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034201.231011_1059.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:36:41.231011Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:02Z" } 2023-09-30T00:37:02.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034201.231011_1059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:36:41.231011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:02.222 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:02.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:02.222 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:37:02.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:02.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:02.223 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:02.266 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:02.266 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:02.337 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[]} 2023-09-30T00:37:02.337 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-85 2023-09-30T00:37:02.337 DEBUG:boto:path=/dmuhsu-85/ 2023-09-30T00:37:02.338 DEBUG:boto:auth_path=/dmuhsu-85/ 2023-09-30T00:37:02.338 DEBUG:boto:Method: PUT 2023-09-30T00:37:02.338 DEBUG:boto:Path: /dmuhsu-85/ 2023-09-30T00:37:02.338 DEBUG:boto:Data: 2023-09-30T00:37:02.338 DEBUG:boto:Headers: {} 2023-09-30T00:37:02.338 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:02.338 DEBUG:boto:Port: 8000 2023-09-30T00:37:02.338 DEBUG:boto:Params: {} 2023-09-30T00:37:02.339 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:37:02.339 DEBUG:boto:Token: None 2023-09-30T00:37:02.339 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:37:02 GMT /dmuhsu-85/ 2023-09-30T00:37:02.339 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FKD/l7LMHiS+T9mHn/VyxGdE6bI= 2023-09-30T00:37:02.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FKD/l7LMHiS+T9mHn/VyxGdE6bI=', 'Content-Length': '0'} 2023-09-30T00:37:02.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b215159a44351f7-0065176dae-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:02.371 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-86 2023-09-30T00:37:02.371 DEBUG:boto:path=/dmuhsu-86/ 2023-09-30T00:37:02.371 DEBUG:boto:auth_path=/dmuhsu-86/ 2023-09-30T00:37:02.372 DEBUG:boto:Method: PUT 2023-09-30T00:37:02.372 DEBUG:boto:Path: /dmuhsu-86/ 2023-09-30T00:37:02.372 DEBUG:boto:Data: 2023-09-30T00:37:02.372 DEBUG:boto:Headers: {} 2023-09-30T00:37:02.372 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:02.372 DEBUG:boto:Port: 8000 2023-09-30T00:37:02.372 DEBUG:boto:Params: {} 2023-09-30T00:37:02.372 DEBUG:boto:Token: None 2023-09-30T00:37:02.372 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:37:02 GMT /dmuhsu-86/ 2023-09-30T00:37:02.373 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8jBjkCJyCOrzg0N4F4HsZOS9avE= 2023-09-30T00:37:02.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8jBjkCJyCOrzg0N4F4HsZOS9avE=', 'Content-Length': '0'} 2023-09-30T00:37:02.401 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb7bf512a0416496-0065176dae-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:02.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.403 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:02.448 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:02.448 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:02.489 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:37:02.503 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.504 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:02.505 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:37:02.506 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:02.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:37:02.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:02.507 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:02.563 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:02.563 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:02.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:02.597+0000 7f6b69d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:02.598 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:02.598+0000 7f6b69d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:02.613 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.354141_1060.1", 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.354141Z", 2023-09-30T00:37:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:37:02.620 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:02.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.354141_1060.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.354141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:02Z" } 2023-09-30T00:37:02.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.354141_1060.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.354141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:02.621 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:02.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.354141_1060.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:02.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034222.397146_1067.1 target=1_1696034222.354141_1060.1 2023-09-30T00:37:07.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:07.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:07.622 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:07.675 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:07.676 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:07.709 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:07.708+0000 7fadcb736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:07.710 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:07.709+0000 7fadcb736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:07.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:07.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:07.727 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:07.728 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:07.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:07.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:37:07.731 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:37:07.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:07.732 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:37:07.732 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:07.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:07Z" } 2023-09-30T00:37:07.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:07.733 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:07.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:07.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:37:07.734 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:37:07.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:07.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:07.734 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:07.790 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:07.790 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:07.835 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:07.834+0000 7f599d929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:07.836 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:07.835+0000 7f599d929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:07.856 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:37:07.856 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:37:07.856 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:37:07.856 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:37:07.856 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:07.857 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.858 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:07.859 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:07.860 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:07.861 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:07.862 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:07.863 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:37:07.864 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:37:07.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:07Z" } 2023-09-30T00:37:07.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:07.865 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:07.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:07.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:37:07.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:07.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:07.866 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:07.912 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:07.913 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:07.978 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:37:07.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:07.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:07.978 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:08.027 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.027 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.075 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:37:08.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-85', '--dest-bucket=dmuhsu-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-85', '--dest-bucket=dmuhsu-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.076 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-85 --dest-bucket=dmuhsu-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:08.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.187 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:37:08.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.188 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:08.233 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.233 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.309 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:37:08.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.311 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:08.356 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.356 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 17, 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:37:08.408 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.409 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:37:08.410 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:37:08.411 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.412 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:37:08.413 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:37:08.414 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:37:08.415 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:37:08.416 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:37:08.417 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:37:08.418 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:37:08.427 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:37:08.428 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.429 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.429 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.429 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-85", 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-86", 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:37:08.431 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.432 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:37:08.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.434 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:37:08.434 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:37:08.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:37:08.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.436 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:08.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.436 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:37:08.436 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:37:08.436 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:37:08.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:37:08.438 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:37:08.439 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:37:08.440 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:08.441 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:37:08.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:37:08.443 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:37:08.485 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.485 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.576 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:37:08.577 DEBUG:boto:path=/dmuhsu-85/ 2023-09-30T00:37:08.577 DEBUG:boto:auth_path=/dmuhsu-85/ 2023-09-30T00:37:08.577 DEBUG:boto:Method: HEAD 2023-09-30T00:37:08.577 DEBUG:boto:Path: /dmuhsu-85/ 2023-09-30T00:37:08.577 DEBUG:boto:Data: 2023-09-30T00:37:08.578 DEBUG:boto:Headers: {} 2023-09-30T00:37:08.578 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:08.578 DEBUG:boto:Port: 8000 2023-09-30T00:37:08.578 DEBUG:boto:Params: {} 2023-09-30T00:37:08.578 DEBUG:boto:Token: None 2023-09-30T00:37:08.578 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:37:08 GMT /dmuhsu-85/ 2023-09-30T00:37:08.579 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg= 2023-09-30T00:37:08.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg=', 'Content-Length': '0'} 2023-09-30T00:37:08.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054a8ecc3e9fe1a17-0065176db4-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:08.692 DEBUG:boto:path=/dmuhsu-85/obj1 2023-09-30T00:37:08.693 DEBUG:boto:auth_path=/dmuhsu-85/obj1 2023-09-30T00:37:08.693 DEBUG:boto:Method: PUT 2023-09-30T00:37:08.693 DEBUG:boto:Path: /dmuhsu-85/obj1 2023-09-30T00:37:08.693 DEBUG:boto:Data: 2023-09-30T00:37:08.694 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:37:08.694 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:08.694 DEBUG:boto:Port: 8000 2023-09-30T00:37:08.694 DEBUG:boto:Params: {} 2023-09-30T00:37:08.694 DEBUG:boto:Token: None 2023-09-30T00:37:08.694 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:37:08 GMT /dmuhsu-85/obj1 2023-09-30T00:37:08.694 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:HkHgE8LO3YC73om3Zwp6NlOOqYc= 2023-09-30T00:37:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HkHgE8LO3YC73om3Zwp6NlOOqYc='} 2023-09-30T00:37:08.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e1d3b642a0b012-0065176db4-2658-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:08.722 DEBUG:boto:path=/dmuhsu-85/ 2023-09-30T00:37:08.722 DEBUG:boto:auth_path=/dmuhsu-85/ 2023-09-30T00:37:08.722 DEBUG:boto:Method: HEAD 2023-09-30T00:37:08.722 DEBUG:boto:Path: /dmuhsu-85/ 2023-09-30T00:37:08.722 DEBUG:boto:Data: 2023-09-30T00:37:08.722 DEBUG:boto:Headers: {} 2023-09-30T00:37:08.722 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:08.723 DEBUG:boto:Port: 8000 2023-09-30T00:37:08.723 DEBUG:boto:Params: {} 2023-09-30T00:37:08.723 DEBUG:boto:Token: None 2023-09-30T00:37:08.723 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:37:08 GMT /dmuhsu-85/ 2023-09-30T00:37:08.723 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg= 2023-09-30T00:37:08.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg=', 'Content-Length': '0'} 2023-09-30T00:37:08.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdc36e07fbcf49dc-0065176db4-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:08.726 DEBUG:boto:path=/dmuhsu-85/obj2 2023-09-30T00:37:08.726 DEBUG:boto:auth_path=/dmuhsu-85/obj2 2023-09-30T00:37:08.727 DEBUG:boto:Method: PUT 2023-09-30T00:37:08.727 DEBUG:boto:Path: /dmuhsu-85/obj2 2023-09-30T00:37:08.727 DEBUG:boto:Data: 2023-09-30T00:37:08.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:37:08.727 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:37:08.727 DEBUG:boto:Port: 8000 2023-09-30T00:37:08.727 DEBUG:boto:Params: {} 2023-09-30T00:37:08.727 DEBUG:boto:Token: None 2023-09-30T00:37:08.727 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:37:08 GMT /dmuhsu-85/obj2 2023-09-30T00:37:08.728 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:avICBwO7NODE2Jiju4myZgUB2o4= 2023-09-30T00:37:08.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:avICBwO7NODE2Jiju4myZgUB2o4='} 2023-09-30T00:37:08.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f25d9f4d77987fa4-0065176db4-2658-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:37:08.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:08.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:08.775 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:08.837 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:08.838 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:08.932 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:37:08.991 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:37:08.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.991 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:08.991 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:08.992 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:08.993 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:08.993 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:37:08.993 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:08.993 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:37:08.994 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:08.994 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:37:08.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:08.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:08.994 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:09.041 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:09.042 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:09.071 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:09.070+0000 7f3af1024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:09.072 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:37:09.071+0000 7f3af1024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:09.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:09.085 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:09.089 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:09.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:37:09.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:09.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:09Z" } 2023-09-30T00:37:09.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:09.093 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:09.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:09.094 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:37:09.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:37:09.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.094 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:09.157 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:09.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:09.243 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:09.243+0000 7f6ebc136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:09.244 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:37:09.244+0000 7f6ebc136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:37:09.285 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:37:09.285 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:37:09.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:09.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:09.288 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:09.289 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:37:09.290 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.291 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:37:09.292 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:37:09.292 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:37:09.292 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.292 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:37:09.292 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:37:09.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:37:09Z" } 2023-09-30T00:37:09.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:37:09.293 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:09.293 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:37:09.294 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:37:09.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.294 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:09.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:09.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:09.493 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:37:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.493 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.494 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:37:09.495 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:37:09.496 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:09.496 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:37:09.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:09.497 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:09.545 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:09.591 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:09.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:09.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:37:09.597 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:09.598 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:09Z" } 2023-09-30T00:37:09.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:09.599 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:09.599 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:09.599 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:14.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:14.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:14.601 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:14.656 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:14.656 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:14.705 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:14Z" 2023-09-30T00:37:14.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:14.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:14Z" } 2023-09-30T00:37:14.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:14.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:14.713 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:14.713 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:19.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:19.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:19.715 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:19.769 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:19.769 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:19.823 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:19.824 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:19.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:19.827 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:19.828 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:19.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:19.830 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:19.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:19.830 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:19Z" 2023-09-30T00:37:19.830 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:19.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:19Z" } 2023-09-30T00:37:19.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:19.831 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:19.831 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:19.831 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:24.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:24.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:24.833 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:24.888 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:24.888 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:24.939 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:24.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:24.941 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:24.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:24.943 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:24.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:24.945 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:24.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:24.945 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:24Z" 2023-09-30T00:37:24.945 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:24.946 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:24Z" } 2023-09-30T00:37:24.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:24.946 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:24.947 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:24.947 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:29.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:29.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:29.948 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:30.002 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:30.002 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:30.051 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:30.051 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:30.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:30.053 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:30.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:30.055 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:30Z" 2023-09-30T00:37:30.057 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:30.058 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:30Z" } 2023-09-30T00:37:30.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:30.059 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:30.059 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:30.059 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:35.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:35.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:35.061 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:35.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:35.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:35.169 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:35.170 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:35.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:35.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:35.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:35Z" 2023-09-30T00:37:35.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:35.176 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:35Z" } 2023-09-30T00:37:35.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:35.177 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:35.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:35.177 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:40.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:40.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:40.179 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:40.233 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:40.233 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:40.284 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:40.284 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:40.284 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:40.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:40.284 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:40.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:40.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:40.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:40.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:40Z" 2023-09-30T00:37:40.290 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:40.291 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:40Z" } 2023-09-30T00:37:40.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:40.292 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:40.292 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:40.292 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:45.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:45.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:45.293 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:45.346 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:45.346 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:45.395 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:45.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:45.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:45.398 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:45.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:45.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:45Z" 2023-09-30T00:37:45.401 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:45.402 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:45Z" } 2023-09-30T00:37:45.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:45.402 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:45.402 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:45.403 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:50.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:50.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:50.404 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:50.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:50.463 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.510 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:50.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:50.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:50.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:50.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:50.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:50.516 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:50Z" 2023-09-30T00:37:50.516 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:50.516 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:50Z" } 2023-09-30T00:37:50.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:50.517 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:50.517 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:50.517 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:37:55.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:55.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:37:55.519 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:37:55.574 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:37:55.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:37:55.624 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:37:55.624 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:55.625 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:37:55.626 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:55.627 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:37:55.629 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:37:55Z" 2023-09-30T00:37:55.630 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:37:55.631 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:37:55Z" } 2023-09-30T00:37:55.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:37:55.631 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:37:55.632 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:37:55.632 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:00.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:00.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:00.633 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:00.688 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:00.688 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:00.738 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:00.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:00.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:00.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:00.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:00.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:00.744 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:00Z" 2023-09-30T00:38:00.745 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:00.745 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:00Z" } 2023-09-30T00:38:00.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:00.746 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:00.746 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:00.746 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:05.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:05.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:05.748 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:05.802 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:05.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:05.853 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:05.854 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:05.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:05.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:05Z" 2023-09-30T00:38:05.859 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:05.860 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:05Z" } 2023-09-30T00:38:05.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:05.860 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:05.861 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:05.861 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:10.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:10.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:10.862 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:10.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:10.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:10.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:10.968 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:10.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:10.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:10.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:10.970 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:10.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:10.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:10Z" 2023-09-30T00:38:10.974 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:10.975 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:10Z" } 2023-09-30T00:38:10.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:10.976 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:10.976 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:10.976 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:15.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:15.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:15.978 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:16.033 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:16.033 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.083 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:16.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:16.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:16.086 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:16.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:16Z" 2023-09-30T00:38:16.088 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:16.089 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:16Z" } 2023-09-30T00:38:16.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:16.090 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:16.090 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:16.090 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:21.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:21.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:21.091 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:21.146 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:21.147 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:21.201 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:21.201 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:21.201 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:21.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:21.201 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:21.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:21.203 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:21Z" 2023-09-30T00:38:21.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:21.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:21Z" } 2023-09-30T00:38:21.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:21.214 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:21.214 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:21.214 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:26.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:26.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:26.216 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:26.270 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:26.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:26.320 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:26Z" 2023-09-30T00:38:26.326 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:26.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:26Z" } 2023-09-30T00:38:26.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:26.328 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:26.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:26.328 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:31.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:31.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:31.329 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:31.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:31.385 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:31.435 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:31Z" 2023-09-30T00:38:31.441 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:31.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:31Z" } 2023-09-30T00:38:31.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:31.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:31.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:31.443 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:36.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:36.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:36.445 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:36.500 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:36.500 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:36.550 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:36.551 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:36.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:38:36.553 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:36.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:36.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:36.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:36.557 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:36Z" 2023-09-30T00:38:36.557 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:36.558 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024227", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:34:37.712703Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.234852Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:36Z" } 2023-09-30T00:38:36.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:38:36.558 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:36.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:38:36.558 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:38:41.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.560 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:41.615 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:41.615 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:41.665 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:41.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:38:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:38:41.672 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:41.673 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:38:41Z" } 2023-09-30T00:38:41.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:38:41.674 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:41.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:38:41.674 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:38:41.674 DEBUG:boto:path=/dmuhsu-85/ 2023-09-30T00:38:41.675 DEBUG:boto:auth_path=/dmuhsu-85/ 2023-09-30T00:38:41.675 DEBUG:boto:Method: HEAD 2023-09-30T00:38:41.675 DEBUG:boto:Path: /dmuhsu-85/ 2023-09-30T00:38:41.675 DEBUG:boto:Data: 2023-09-30T00:38:41.675 DEBUG:boto:Headers: {} 2023-09-30T00:38:41.675 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:38:41.675 DEBUG:boto:Port: 8000 2023-09-30T00:38:41.675 DEBUG:boto:Params: {} 2023-09-30T00:38:41.676 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:38:41.676 DEBUG:boto:Token: None 2023-09-30T00:38:41.676 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:38:41 GMT /dmuhsu-85/ 2023-09-30T00:38:41.676 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mPQAIgjv9iHvkgLlFQlsHOZfIOM= 2023-09-30T00:38:41.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mPQAIgjv9iHvkgLlFQlsHOZfIOM=', 'Content-Length': '0'} 2023-09-30T00:38:41.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aa844018f712c65-0065176e11-1f62-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:41.682 DEBUG:boto:path=/dmuhsu-85/obj1 2023-09-30T00:38:41.682 DEBUG:boto:auth_path=/dmuhsu-85/obj1 2023-09-30T00:38:41.683 DEBUG:boto:Method: HEAD 2023-09-30T00:38:41.683 DEBUG:boto:Path: /dmuhsu-85/obj1 2023-09-30T00:38:41.683 DEBUG:boto:Data: 2023-09-30T00:38:41.683 DEBUG:boto:Headers: {} 2023-09-30T00:38:41.683 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:38:41.683 DEBUG:boto:Port: 8000 2023-09-30T00:38:41.683 DEBUG:boto:Params: {} 2023-09-30T00:38:41.683 DEBUG:boto:Token: None 2023-09-30T00:38:41.683 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:38:41 GMT /dmuhsu-85/obj1 2023-09-30T00:38:41.684 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6/ygMYOt41dKMg54L2Is1g6Y/GU= 2023-09-30T00:38:41.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6/ygMYOt41dKMg54L2Is1g6Y/GU=', 'Content-Length': '0'} 2023-09-30T00:38:41.685 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a873de4c7c422197-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:41.685 DEBUG:boto:path=/dmuhsu-85/obj2 2023-09-30T00:38:41.685 DEBUG:boto:auth_path=/dmuhsu-85/obj2 2023-09-30T00:38:41.686 DEBUG:boto:Method: HEAD 2023-09-30T00:38:41.686 DEBUG:boto:Path: /dmuhsu-85/obj2 2023-09-30T00:38:41.686 DEBUG:boto:Data: 2023-09-30T00:38:41.686 DEBUG:boto:Headers: {} 2023-09-30T00:38:41.686 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:38:41.686 DEBUG:boto:Port: 8000 2023-09-30T00:38:41.686 DEBUG:boto:Params: {} 2023-09-30T00:38:41.686 DEBUG:boto:Token: None 2023-09-30T00:38:41.686 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:38:41 GMT /dmuhsu-85/obj2 2023-09-30T00:38:41.687 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:K6mXwRXwXSISX5VgJSGjWw6FvYg= 2023-09-30T00:38:41.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:K6mXwRXwXSISX5VgJSGjWw6FvYg=', 'Content-Length': '0'} 2023-09-30T00:38:41.688 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000084504d5956cd39fc-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:41.688 DEBUG:boto:path=/dmuhsu-86/ 2023-09-30T00:38:41.688 DEBUG:boto:auth_path=/dmuhsu-86/ 2023-09-30T00:38:41.689 DEBUG:boto:Method: HEAD 2023-09-30T00:38:41.689 DEBUG:boto:Path: /dmuhsu-86/ 2023-09-30T00:38:41.689 DEBUG:boto:Data: 2023-09-30T00:38:41.689 DEBUG:boto:Headers: {} 2023-09-30T00:38:41.689 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:38:41.689 DEBUG:boto:Port: 8000 2023-09-30T00:38:41.689 DEBUG:boto:Params: {} 2023-09-30T00:38:41.689 DEBUG:boto:Token: None 2023-09-30T00:38:41.690 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:38:41 GMT /dmuhsu-86/ 2023-09-30T00:38:41.690 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QtdIw2gSXEfqr3wktxZvzuzhT7M= 2023-09-30T00:38:41.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QtdIw2gSXEfqr3wktxZvzuzhT7M=', 'Content-Length': '0'} 2023-09-30T00:38:41.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032834f77cba017cc-0065176e11-1f62-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:41.692 DEBUG:boto:path=/dmuhsu-86/obj1 2023-09-30T00:38:41.692 DEBUG:boto:auth_path=/dmuhsu-86/obj1 2023-09-30T00:38:41.693 DEBUG:boto:Method: HEAD 2023-09-30T00:38:41.693 DEBUG:boto:Path: /dmuhsu-86/obj1 2023-09-30T00:38:41.693 DEBUG:boto:Data: 2023-09-30T00:38:41.693 DEBUG:boto:Headers: {} 2023-09-30T00:38:41.693 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:38:41.693 DEBUG:boto:Port: 8000 2023-09-30T00:38:41.693 DEBUG:boto:Params: {} 2023-09-30T00:38:41.693 DEBUG:boto:Token: None 2023-09-30T00:38:41.693 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:38:41 GMT /dmuhsu-86/obj1 2023-09-30T00:38:41.694 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ndzHa0U2dnJaPfqkvy0rSdgKjwc= 2023-09-30T00:38:41.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ndzHa0U2dnJaPfqkvy0rSdgKjwc=', 'Content-Length': '0'} 2023-09-30T00:38:41.695 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cb081bf9d6ac6503-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:41.698 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2023-09-30T00:38:41.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.702 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:41.754 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:41.754 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:41.807 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:41.826 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:41.827 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:38:41.828 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:41.828 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:38:41.829 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:41.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:38:41.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.829 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:41.877 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:41.877 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:41.907 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:41.906+0000 7f8c5ec26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:41.908 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:41.907+0000 7f8c5ec26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:41.921 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:41.921 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:41.921 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:41.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:41.921 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:41.922 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.923 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:41.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:41.926 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:41.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:41.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:38:41.930 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:41.932 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:41Z" } 2023-09-30T00:38:41.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:41.932 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:41.932 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:41.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:38:41.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:38:41.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:41.934 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:41.980 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:41.981 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:42.028 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:38:42.028+0000 7f63ff017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:42.029 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:38:42.029+0000 7f63ff017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:42.047 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:38:42.047 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.048 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:38:42.050 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:38:42.051 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:38:42.052 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.053 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:38:42.054 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:38:42.054 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:38:42.054 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.054 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:38:42.054 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:38:42.056 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:42Z" } 2023-09-30T00:38:42.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:42.057 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:42.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:42.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:38:42.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:38:42.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:38:42.058 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:38:42.100 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:42.100 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:42.151 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:38:42.151 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-87 2023-09-30T00:38:42.152 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:38:42.152 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:38:42.152 DEBUG:boto:Method: PUT 2023-09-30T00:38:42.152 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:38:42.152 DEBUG:boto:Data: 2023-09-30T00:38:42.152 DEBUG:boto:Headers: {} 2023-09-30T00:38:42.152 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:38:42.153 DEBUG:boto:Port: 8000 2023-09-30T00:38:42.153 DEBUG:boto:Params: {} 2023-09-30T00:38:42.153 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:38:42.153 DEBUG:boto:Token: None 2023-09-30T00:38:42.153 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:38:42 GMT /dmuhsu-87/ 2023-09-30T00:38:42.153 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:SpbZ5E5cM00GG/T7vOQu7WlctIQ= 2023-09-30T00:38:42.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SpbZ5E5cM00GG/T7vOQu7WlctIQ=', 'Content-Length': '0'} 2023-09-30T00:38:42.170 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000510d2902a9cb7bf4-0065176e12-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:42.171 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-88 2023-09-30T00:38:42.171 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:38:42.171 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:38:42.171 DEBUG:boto:Method: PUT 2023-09-30T00:38:42.171 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:38:42.171 DEBUG:boto:Data: 2023-09-30T00:38:42.172 DEBUG:boto:Headers: {} 2023-09-30T00:38:42.172 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:38:42.172 DEBUG:boto:Port: 8000 2023-09-30T00:38:42.172 DEBUG:boto:Params: {} 2023-09-30T00:38:42.172 DEBUG:boto:Token: None 2023-09-30T00:38:42.172 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:38:42 GMT /dmuhsu-88/ 2023-09-30T00:38:42.172 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Eof8SRN4DPl/AamT6OB0sWJ51Uw= 2023-09-30T00:38:42.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eof8SRN4DPl/AamT6OB0sWJ51Uw=', 'Content-Length': '0'} 2023-09-30T00:38:42.188 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002404ae5f4e95acb3-0065176e12-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:38:42.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:42.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:42.189 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:42.235 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:42.235 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:42.269 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:38:42.286 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:38:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:38:42.186786Z" 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:38:42.288 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:38:42.289 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:42.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:38:42.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:42.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:42.290 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:42.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:42.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:42.368 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:42.367+0000 7fbdf551e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:42.368 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:42.368+0000 7fbdf551e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:42.382 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:42.382 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:42.382 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:42.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:42.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:42.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:42.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:38:42.388 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:38:42.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:42.389 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:38:42.389 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:42.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:42Z" } 2023-09-30T00:38:42.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:42.390 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:42.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:42.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:38:47.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:47.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:47.392 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:47.445 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:47.445 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:47.479 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:47.478+0000 7f12b7729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:47.480 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:47.479+0000 7f12b7729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:47.494 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:47.495 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:47.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:47.497 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:47.498 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:47.499 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:47.500 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:47.500 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:47.500 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:47.500 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:47.500 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:47.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:38:47.502 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:38:47.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:47.502 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:47Z" 2023-09-30T00:38:47.502 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:47.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:47Z" } 2023-09-30T00:38:47.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:47.504 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:47.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:47.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:38:52.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:52.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:52.505 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:52.559 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:52.559 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:52.592 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:52.591+0000 7f504540a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:52.593 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:52.592+0000 7f504540a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:52.608 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:52.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:52.610 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:52.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:52.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:52.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:52.614 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:52Z" 2023-09-30T00:38:52.615 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:52.615 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:52Z" } 2023-09-30T00:38:52.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:52.616 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:52.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:52.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:38:57.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:57.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:38:57.617 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:38:57.670 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:38:57.671 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:38:57.705 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:57.704+0000 7f9285333880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:57.706 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:38:57.705+0000 7f9285333880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:38:57.720 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:38:57.720 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:38:57.721 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:57.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:57.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:38:57.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:38:57.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:38:57.726 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:57.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:38:57Z" 2023-09-30T00:38:57.728 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:38:57.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:38:57Z" } 2023-09-30T00:38:57.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:38:57.730 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:57.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:38:57.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:39:02.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:02.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:02.732 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:02.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:02.818+0000 7fe93e80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:02.819 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:02.819+0000 7fe93e80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:02.834 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:39:02.836 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:02.837 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:39:02.839 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:02.840 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:02.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.842 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:02.842 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:39:02.842 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:39:02.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:02.843 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:02Z" 2023-09-30T00:39:02.844 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:02.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:02Z" } 2023-09-30T00:39:02.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:02.846 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:02.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:02.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:39:07.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:07.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:07.847 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:07.900 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:07.901 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:07.934 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:07.934+0000 7f8ca1414880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:07.935 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:07.935+0000 7f8ca1414880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:07.949 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:07.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:07.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:07.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:07.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:07.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:07.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:07Z" 2023-09-30T00:39:07.956 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:07.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034222.397146_1067.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:02.397146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:07Z" } 2023-09-30T00:39:07.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:07.959 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:07.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:07.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:39:12.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:12.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:12.961 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:13.014 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:13.014 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:13.048 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:13.047+0000 7fd324d27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:13.048 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:13.048+0000 7fd324d27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:13.063 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:13.063 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:13.064 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:13.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:13.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.163175_1099.1", 2023-09-30T00:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.163175Z", 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:13.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:13.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:13Z" 2023-09-30T00:39:13.070 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:13.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.163175_1099.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.163175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:13Z" } 2023-09-30T00:39:13.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.163175_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.163175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:13.076 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:13.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.163175_1099.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:13.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034322.163175_1099.1 2023-09-30T00:39:18.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:18.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:18.082 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.135 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.168 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:18.167+0000 7f73edd1c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:18.169 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:18.169+0000 7f73edd1c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:18.184 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:18.184 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:18.184 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:18.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:18.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:18.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:18.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:18.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:39:18.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:18.191 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:39:18.191 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:18.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:18Z" } 2023-09-30T00:39:18.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:18.192 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:18.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:18.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:39:18.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:39:18.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:18.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:18.193 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:18.247 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.247 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.281 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:39:18.279+0000 7f3000a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:18.282 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:39:18.280+0000 7f3000a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:18.296 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:39:18.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.298 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:18.299 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:18.300 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:18.301 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:39:18.302 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:39:18.303 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:39:18.303 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.303 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:39:18.303 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:39:18.304 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:18Z" } 2023-09-30T00:39:18.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:18.304 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:18.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:18.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:39:18.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.305 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.351 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.351 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.402 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:39:18.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.403 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.453 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.508 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:39:18.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-87', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-87', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.509 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-87 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-87 --dest-bucket=dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.559 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.559 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.635 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:39:18.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-88', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-88', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.635 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-88 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-88 --dest-bucket=dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.686 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.686 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.735 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:39:18.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.735 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.780 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.781 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.841 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:39:18.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.842 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:18.886 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:18.887 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:18.936 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 18, 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:39:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:39:18.940 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:39:18.941 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:39:18.942 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:39:18.943 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:39:18.944 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:39:18.945 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.946 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:39:18.947 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.948 INFO:teuthology.orchestra.run.smithi175.stdout: "directional": [ 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "source_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "dest_zone": "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.949 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-85", 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-86", 2023-09-30T00:39:18.950 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:39:18.951 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-87", 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-87", 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.952 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-88", 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:39:18.953 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.954 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-88", 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-88", 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-88", 2023-09-30T00:39:18.955 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:39:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:39:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.956 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:39:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:39:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:39:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:39:18.957 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:39:18.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.959 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.960 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:39:18.961 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:39:18.962 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:39:18.963 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:18.964 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:39:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:39:18.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:18.965 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:39:18.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:39:18.966 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:39:19.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:19.007 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:19.082 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:39:19.082 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:39:19.083 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:39:19.083 DEBUG:boto:Method: HEAD 2023-09-30T00:39:19.084 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:39:19.084 DEBUG:boto:Data: 2023-09-30T00:39:19.084 DEBUG:boto:Headers: {} 2023-09-30T00:39:19.084 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.085 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.085 DEBUG:boto:Params: {} 2023-09-30T00:39:19.085 DEBUG:boto:Token: None 2023-09-30T00:39:19.086 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-87/ 2023-09-30T00:39:19.086 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw= 2023-09-30T00:39:19.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw=', 'Content-Length': '0'} 2023-09-30T00:39:19.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd25b6f0398fdcb-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.326 DEBUG:boto:path=/dmuhsu-87/obj1 2023-09-30T00:39:19.326 DEBUG:boto:auth_path=/dmuhsu-87/obj1 2023-09-30T00:39:19.327 DEBUG:boto:Method: PUT 2023-09-30T00:39:19.327 DEBUG:boto:Path: /dmuhsu-87/obj1 2023-09-30T00:39:19.327 DEBUG:boto:Data: 2023-09-30T00:39:19.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:39:19.327 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.327 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.327 DEBUG:boto:Params: {} 2023-09-30T00:39:19.327 DEBUG:boto:Token: None 2023-09-30T00:39:19.328 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-87/obj1 2023-09-30T00:39:19.328 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:1tQk14w0DntvCXU9mUX9I926QEY= 2023-09-30T00:39:19.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tQk14w0DntvCXU9mUX9I926QEY='} 2023-09-30T00:39:19.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b93dc08825624c28-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.368 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:39:19.368 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:39:19.368 DEBUG:boto:Method: HEAD 2023-09-30T00:39:19.368 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:39:19.368 DEBUG:boto:Data: 2023-09-30T00:39:19.368 DEBUG:boto:Headers: {} 2023-09-30T00:39:19.368 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.368 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.368 DEBUG:boto:Params: {} 2023-09-30T00:39:19.369 DEBUG:boto:Token: None 2023-09-30T00:39:19.369 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-87/ 2023-09-30T00:39:19.369 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw= 2023-09-30T00:39:19.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw=', 'Content-Length': '0'} 2023-09-30T00:39:19.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6edd77e9608cee4-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.374 DEBUG:boto:path=/dmuhsu-87/obj2 2023-09-30T00:39:19.374 DEBUG:boto:auth_path=/dmuhsu-87/obj2 2023-09-30T00:39:19.374 DEBUG:boto:Method: PUT 2023-09-30T00:39:19.374 DEBUG:boto:Path: /dmuhsu-87/obj2 2023-09-30T00:39:19.374 DEBUG:boto:Data: 2023-09-30T00:39:19.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:39:19.375 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.375 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.375 DEBUG:boto:Params: {} 2023-09-30T00:39:19.375 DEBUG:boto:Token: None 2023-09-30T00:39:19.375 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-87/obj2 2023-09-30T00:39:19.375 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:H9ZF2ZRYR4nXJa7hLuPIC7jvcho= 2023-09-30T00:39:19.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H9ZF2ZRYR4nXJa7hLuPIC7jvcho='} 2023-09-30T00:39:19.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000526e2252af677e7e-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.419 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:39:19.419 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:39:19.419 DEBUG:boto:Method: HEAD 2023-09-30T00:39:19.420 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:39:19.420 DEBUG:boto:Data: 2023-09-30T00:39:19.420 DEBUG:boto:Headers: {} 2023-09-30T00:39:19.420 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.420 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.420 DEBUG:boto:Params: {} 2023-09-30T00:39:19.420 DEBUG:boto:Token: None 2023-09-30T00:39:19.421 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-88/ 2023-09-30T00:39:19.421 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU= 2023-09-30T00:39:19.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU=', 'Content-Length': '0'} 2023-09-30T00:39:19.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9a46fbb2eb7b840-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.428 DEBUG:boto:path=/dmuhsu-88/obj3 2023-09-30T00:39:19.428 DEBUG:boto:auth_path=/dmuhsu-88/obj3 2023-09-30T00:39:19.429 DEBUG:boto:Method: PUT 2023-09-30T00:39:19.429 DEBUG:boto:Path: /dmuhsu-88/obj3 2023-09-30T00:39:19.429 DEBUG:boto:Data: 2023-09-30T00:39:19.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:39:19.430 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.430 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.431 DEBUG:boto:Params: {} 2023-09-30T00:39:19.431 DEBUG:boto:Token: None 2023-09-30T00:39:19.431 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-88/obj3 2023-09-30T00:39:19.432 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2tDF0guenHidnmhEZKXyosteqTI= 2023-09-30T00:39:19.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2tDF0guenHidnmhEZKXyosteqTI='} 2023-09-30T00:39:19.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000892ecedf1b6641b7-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.459 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:39:19.459 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:39:19.459 DEBUG:boto:Method: HEAD 2023-09-30T00:39:19.460 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:39:19.460 DEBUG:boto:Data: 2023-09-30T00:39:19.460 DEBUG:boto:Headers: {} 2023-09-30T00:39:19.460 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.460 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.460 DEBUG:boto:Params: {} 2023-09-30T00:39:19.460 DEBUG:boto:Token: None 2023-09-30T00:39:19.461 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-88/ 2023-09-30T00:39:19.461 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU= 2023-09-30T00:39:19.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU=', 'Content-Length': '0'} 2023-09-30T00:39:19.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e1dc089ae220f7f-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.465 DEBUG:boto:path=/dmuhsu-88/obj4 2023-09-30T00:39:19.465 DEBUG:boto:auth_path=/dmuhsu-88/obj4 2023-09-30T00:39:19.465 DEBUG:boto:Method: PUT 2023-09-30T00:39:19.466 DEBUG:boto:Path: /dmuhsu-88/obj4 2023-09-30T00:39:19.466 DEBUG:boto:Data: 2023-09-30T00:39:19.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:39:19.466 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:39:19.467 DEBUG:boto:Port: 8000 2023-09-30T00:39:19.467 DEBUG:boto:Params: {} 2023-09-30T00:39:19.467 DEBUG:boto:Token: None 2023-09-30T00:39:19.467 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:39:19 GMT /dmuhsu-88/obj4 2023-09-30T00:39:19.467 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:n71fatriBdqpSdSf/MhFwqqLSMY= 2023-09-30T00:39:19.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n71fatriBdqpSdSf/MhFwqqLSMY='} 2023-09-30T00:39:19.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e09249a3c60b927-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:39:19.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.511 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:19.580 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:19.580 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:19.677 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:39:19.716 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:39:19.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.716 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:19.716 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:39:19.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:38:42.186786Z" 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.717 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:19.718 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:39:19.718 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.718 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:39:19.719 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:19.719 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:39:19.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.720 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:19.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:19.795 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:19.794+0000 7fbeb400f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:19.796 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:39:19.795+0000 7fbeb400f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:19.810 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:19.811 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:19.812 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:19.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:19.815 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:39:19.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:19.817 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:39:19.817 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:19.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:19Z" } 2023-09-30T00:39:19.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:19.818 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:19.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:19.818 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:39:19.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:39:19.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.819 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:19.877 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:19.878 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:19.941 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:39:19.939+0000 7fb0ad726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:19.942 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:39:19.940+0000 7fb0ad726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:19.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:19.966 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:19.967 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:39:19.968 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:39:19.969 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.970 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:39:19.971 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:39:19.972 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:39:19Z" } 2023-09-30T00:39:19.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:39:19.973 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:19.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:39:19.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:39:19.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:19.973 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:20.024 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:20.024 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:20.134 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:39:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:20.134 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:39:20.135 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:39:20.136 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:39:20.138 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:20.138 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:39:20.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:20.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:20.138 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:20.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:20.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.238 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:20.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:20.238 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:20.238 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:20.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:20.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:20.240 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:20.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:20.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:20.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:20.245 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:20.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:20.245 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:20Z" 2023-09-30T00:39:20.245 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:20.246 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:20Z" } 2023-09-30T00:39:20.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:20.247 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:20.247 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:20.247 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:25.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:25.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:25.248 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:25.304 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:25.304 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:25.355 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:25.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:25.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:25.358 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:25.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.360 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:25.361 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:25.361 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:25.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:25.361 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:25Z" 2023-09-30T00:39:25.361 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:25.362 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:25Z" } 2023-09-30T00:39:25.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:25.363 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:25.363 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:25.363 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:30.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:30.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:30.364 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:30.419 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:30.420 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:30.470 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:30.470 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:30.471 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:30.472 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:30.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:30.475 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:30.476 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:30Z" 2023-09-30T00:39:30.477 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:30.477 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:30Z" } 2023-09-30T00:39:30.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:30.478 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:30.478 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:30.478 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:35.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:35.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:35.480 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:35.535 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:35.535 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:35.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:35.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:35.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:35.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:35.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:35.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:35.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:35.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:35.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:35Z" 2023-09-30T00:39:35.594 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:35.595 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:35Z" } 2023-09-30T00:39:35.596 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:35.596 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:35.596 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:35.596 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:40.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:40.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:40.597 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:40.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:40.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:40.703 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:40.703 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:40.704 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:40.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:40.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:40.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:40Z" 2023-09-30T00:39:40.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:40.710 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:40Z" } 2023-09-30T00:39:40.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:40.711 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:40.711 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:40.711 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:45.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:45.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:45.712 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:45.767 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:45.767 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:45.818 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:45.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:45.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:45.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:45.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:45.823 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:45.824 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:45.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:45.825 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:45Z" 2023-09-30T00:39:45.825 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:45.826 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:45Z" } 2023-09-30T00:39:45.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:45.826 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:45.826 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:45.826 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:50.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:50.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:50.828 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:50.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:50.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.938 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:50.939 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:50.939 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:50.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:50.939 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:50Z" 2023-09-30T00:39:50.939 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:50.940 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:50Z" } 2023-09-30T00:39:50.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:50.940 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:50.940 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:50.941 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:39:55.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:55.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:39:55.942 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:39:55.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:39:55.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:39:56.047 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:39:56.047 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:56.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:56.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.050 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:56.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:39:56.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:39:56Z" 2023-09-30T00:39:56.053 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:39:56.054 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:39:56Z" } 2023-09-30T00:39:56.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:39:56.055 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:39:56.055 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:39:56.055 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:01.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:01.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:01.056 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:01.111 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:01.111 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:01.161 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:01Z" 2023-09-30T00:40:01.167 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:01.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:01Z" } 2023-09-30T00:40:01.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:01.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:01.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:01.169 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:06.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:06.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:06.171 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:06.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:06.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:06.275 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:06.275 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:06.275 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:06.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:06.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:06.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:06.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:06.279 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:06.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:06.281 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:06.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:06.281 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:06Z" 2023-09-30T00:40:06.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:06.282 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:06Z" } 2023-09-30T00:40:06.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:06.282 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:06.282 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:06.282 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:11.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:11.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:11.284 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:11.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:11.339 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:11.391 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:11.391 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:11.391 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:11.391 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:11.392 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:11.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:11.394 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:11.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.396 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:11.397 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:11Z" 2023-09-30T00:40:11.397 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:11.398 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:11Z" } 2023-09-30T00:40:11.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:11.398 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:11.398 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:11.398 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:16.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:16.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:16.400 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:16.455 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:16.455 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:16.505 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:16.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.507 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:16.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:16.507 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:16.507 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:16.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:16.508 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:16.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:16.510 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:16.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:16.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:16.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:16.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:16.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:16.512 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:16.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:16.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:16.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:16.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:16.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:16.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:16.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:16.514 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:16.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.514 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:16.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:16.515 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:16.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:16.515 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:16Z" 2023-09-30T00:40:16.516 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:16.517 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:16Z" } 2023-09-30T00:40:16.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:16.518 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:16.518 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:16.518 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:21.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:21.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:21.520 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:21.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:21.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:21.625 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:21.625 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:21.625 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:21.625 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:21.626 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:21.629 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:21.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:21Z" 2023-09-30T00:40:21.631 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:21.632 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:21Z" } 2023-09-30T00:40:21.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:21.632 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:21.632 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:21.633 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:26.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:26.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:26.634 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:26.689 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:26.689 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:26.740 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:26.740 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:26.741 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:26.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:26.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:26Z" 2023-09-30T00:40:26.746 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:26.747 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:26Z" } 2023-09-30T00:40:26.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:26.748 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:26.748 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:26.748 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:31.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:31.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:31.750 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:31.804 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:31.805 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:31.855 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:31.855 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:31.855 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:31.856 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:31.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:31Z" 2023-09-30T00:40:31.862 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:31.863 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:31Z" } 2023-09-30T00:40:31.863 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:31.863 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:31.864 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:31.864 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:36.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:36.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:36.865 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:36.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:36.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:36.970 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:36.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:36.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:36Z" 2023-09-30T00:40:36.976 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:36.977 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:36Z" } 2023-09-30T00:40:36.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:36.978 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:36.978 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:36.978 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:41.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:41.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:41.980 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:42.035 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:42.035 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:42.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:42.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:42.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:42.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:42.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:42.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:42.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:42.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:42.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:42.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:42.091 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:42Z" 2023-09-30T00:40:42.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:42.093 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:42Z" } 2023-09-30T00:40:42.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:42.093 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:42.094 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:42.094 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:47.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:47.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:47.096 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:47.150 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:47.150 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:47.206 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:47.206 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:47.206 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:47.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:47.208 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:47.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:47Z" 2023-09-30T00:40:47.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:47.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013370", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:31:43.024769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024367", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:37:08.718748Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018474", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:37:08.745804Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:47Z" } 2023-09-30T00:40:47.213 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:40:47.213 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:47.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:40:47.213 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:40:52.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:52.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:52.215 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:52.269 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.269 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.320 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:52.320 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:52.320 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:52.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:52.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:52.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:52.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:52.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:52.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:52Z" 2023-09-30T00:40:52.326 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:52.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013510", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:39:19.485460Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024647", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:39:19.452905Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018614", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:39:19.393348Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:40:52Z" } 2023-09-30T00:40:52.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:40:52.328 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:52.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:40:52.328 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:40:52.328 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:40:52.328 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:40:52.328 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.328 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:40:52.329 DEBUG:boto:Data: 2023-09-30T00:40:52.329 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.329 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.329 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.329 DEBUG:boto:Params: {} 2023-09-30T00:40:52.329 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:40:52.329 DEBUG:boto:Token: None 2023-09-30T00:40:52.330 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-87/ 2023-09-30T00:40:52.330 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hhzzcAiiNPWZlBaIVKQ6LIvQgPE= 2023-09-30T00:40:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hhzzcAiiNPWZlBaIVKQ6LIvQgPE=', 'Content-Length': '0'} 2023-09-30T00:40:52.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da07c63280ae923a-0065176e94-204e-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.336 DEBUG:boto:path=/dmuhsu-87/obj1 2023-09-30T00:40:52.336 DEBUG:boto:auth_path=/dmuhsu-87/obj1 2023-09-30T00:40:52.337 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.337 DEBUG:boto:Path: /dmuhsu-87/obj1 2023-09-30T00:40:52.337 DEBUG:boto:Data: 2023-09-30T00:40:52.337 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.337 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.337 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.337 DEBUG:boto:Params: {} 2023-09-30T00:40:52.337 DEBUG:boto:Token: None 2023-09-30T00:40:52.338 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-87/obj1 2023-09-30T00:40:52.338 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:qplxDd7l6XiXpcrCytV3S7OC7wk= 2023-09-30T00:40:52.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qplxDd7l6XiXpcrCytV3S7OC7wk=', 'Content-Length': '0'} 2023-09-30T00:40:52.339 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e599d5c116dd4a6f-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.340 DEBUG:boto:path=/dmuhsu-87/obj2 2023-09-30T00:40:52.340 DEBUG:boto:auth_path=/dmuhsu-87/obj2 2023-09-30T00:40:52.340 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.340 DEBUG:boto:Path: /dmuhsu-87/obj2 2023-09-30T00:40:52.340 DEBUG:boto:Data: 2023-09-30T00:40:52.340 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.340 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.340 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.340 DEBUG:boto:Params: {} 2023-09-30T00:40:52.341 DEBUG:boto:Token: None 2023-09-30T00:40:52.341 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-87/obj2 2023-09-30T00:40:52.341 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Y8midTiPqzUuQxAfmLua2gLzRVA= 2023-09-30T00:40:52.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y8midTiPqzUuQxAfmLua2gLzRVA=', 'Content-Length': '0'} 2023-09-30T00:40:52.342 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000022bb366fd6fd4a6-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.342 DEBUG:boto:path=/dmuhsu-87/obj3 2023-09-30T00:40:52.342 DEBUG:boto:auth_path=/dmuhsu-87/obj3 2023-09-30T00:40:52.343 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.343 DEBUG:boto:Path: /dmuhsu-87/obj3 2023-09-30T00:40:52.343 DEBUG:boto:Data: 2023-09-30T00:40:52.343 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.343 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.343 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.343 DEBUG:boto:Params: {} 2023-09-30T00:40:52.343 DEBUG:boto:Token: None 2023-09-30T00:40:52.343 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-87/obj3 2023-09-30T00:40:52.344 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:d6tbx1XqAv1JqRoqrK9vXvGf9Uk= 2023-09-30T00:40:52.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d6tbx1XqAv1JqRoqrK9vXvGf9Uk=', 'Content-Length': '0'} 2023-09-30T00:40:52.345 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004082988b79d081f1-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.345 DEBUG:boto:path=/dmuhsu-87/obj4 2023-09-30T00:40:52.345 DEBUG:boto:auth_path=/dmuhsu-87/obj4 2023-09-30T00:40:52.345 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.345 DEBUG:boto:Path: /dmuhsu-87/obj4 2023-09-30T00:40:52.346 DEBUG:boto:Data: 2023-09-30T00:40:52.346 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.346 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.346 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.346 DEBUG:boto:Params: {} 2023-09-30T00:40:52.346 DEBUG:boto:Token: None 2023-09-30T00:40:52.346 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-87/obj4 2023-09-30T00:40:52.347 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+klwmmxzxhyrlzdJq/dBhF8P0f8= 2023-09-30T00:40:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+klwmmxzxhyrlzdJq/dBhF8P0f8=', 'Content-Length': '0'} 2023-09-30T00:40:52.348 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d7221e21254b4758-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.349 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:40:52.349 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:40:52.349 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.349 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:40:52.349 DEBUG:boto:Data: 2023-09-30T00:40:52.349 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.349 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.349 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.349 DEBUG:boto:Params: {} 2023-09-30T00:40:52.349 DEBUG:boto:Token: None 2023-09-30T00:40:52.350 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/ 2023-09-30T00:40:52.350 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:8PB+5srOv7Zcljx13TZLwjzi9SY= 2023-09-30T00:40:52.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PB+5srOv7Zcljx13TZLwjzi9SY=', 'Content-Length': '0'} 2023-09-30T00:40:52.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008872d1ae327efdcc-0065176e94-204e-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.352 DEBUG:boto:path=/dmuhsu-88/obj1 2023-09-30T00:40:52.352 DEBUG:boto:auth_path=/dmuhsu-88/obj1 2023-09-30T00:40:52.352 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.353 DEBUG:boto:Path: /dmuhsu-88/obj1 2023-09-30T00:40:52.353 DEBUG:boto:Data: 2023-09-30T00:40:52.353 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.353 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.353 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.353 DEBUG:boto:Params: {} 2023-09-30T00:40:52.353 DEBUG:boto:Token: None 2023-09-30T00:40:52.353 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj1 2023-09-30T00:40:52.353 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:zoZxqcTVahoNJg2sCuuoqrbUwMI= 2023-09-30T00:40:52.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zoZxqcTVahoNJg2sCuuoqrbUwMI=', 'Content-Length': '0'} 2023-09-30T00:40:52.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051a5820fcb6722f8-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.355 DEBUG:boto:path=/dmuhsu-88/obj1 2023-09-30T00:40:52.355 DEBUG:boto:auth_path=/dmuhsu-88/obj1 2023-09-30T00:40:52.356 DEBUG:boto:Method: GET 2023-09-30T00:40:52.356 DEBUG:boto:Path: /dmuhsu-88/obj1 2023-09-30T00:40:52.356 DEBUG:boto:Data: 2023-09-30T00:40:52.356 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.356 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.356 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.356 DEBUG:boto:Params: {} 2023-09-30T00:40:52.356 DEBUG:boto:Token: None 2023-09-30T00:40:52.356 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj1 2023-09-30T00:40:52.357 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:6rjOnxyBhkUhjUnE8wNLsMB/ieA= 2023-09-30T00:40:52.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6rjOnxyBhkUhjUnE8wNLsMB/ieA=', 'Content-Length': '0'} 2023-09-30T00:40:52.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003027ab0e380050bc-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.358 DEBUG:boto:path=/dmuhsu-88/obj2 2023-09-30T00:40:52.359 DEBUG:boto:auth_path=/dmuhsu-88/obj2 2023-09-30T00:40:52.359 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.359 DEBUG:boto:Path: /dmuhsu-88/obj2 2023-09-30T00:40:52.359 DEBUG:boto:Data: 2023-09-30T00:40:52.359 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.359 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.359 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.359 DEBUG:boto:Params: {} 2023-09-30T00:40:52.359 DEBUG:boto:Token: None 2023-09-30T00:40:52.360 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj2 2023-09-30T00:40:52.360 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:BhHvh7zoXYpTgmFOJXFvvDWDsCc= 2023-09-30T00:40:52.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BhHvh7zoXYpTgmFOJXFvvDWDsCc=', 'Content-Length': '0'} 2023-09-30T00:40:52.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014ef7a7e98e87b30-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.361 DEBUG:boto:path=/dmuhsu-88/obj2 2023-09-30T00:40:52.361 DEBUG:boto:auth_path=/dmuhsu-88/obj2 2023-09-30T00:40:52.362 DEBUG:boto:Method: GET 2023-09-30T00:40:52.362 DEBUG:boto:Path: /dmuhsu-88/obj2 2023-09-30T00:40:52.362 DEBUG:boto:Data: 2023-09-30T00:40:52.362 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.362 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.362 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.362 DEBUG:boto:Params: {} 2023-09-30T00:40:52.362 DEBUG:boto:Token: None 2023-09-30T00:40:52.362 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj2 2023-09-30T00:40:52.363 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:K0ERW6p44nP95mF136gmHv+bJGo= 2023-09-30T00:40:52.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:K0ERW6p44nP95mF136gmHv+bJGo=', 'Content-Length': '0'} 2023-09-30T00:40:52.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d9506aeef81343b-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.406 DEBUG:boto:path=/dmuhsu-88/obj3 2023-09-30T00:40:52.407 DEBUG:boto:auth_path=/dmuhsu-88/obj3 2023-09-30T00:40:52.407 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.407 DEBUG:boto:Path: /dmuhsu-88/obj3 2023-09-30T00:40:52.407 DEBUG:boto:Data: 2023-09-30T00:40:52.407 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.407 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.407 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.407 DEBUG:boto:Params: {} 2023-09-30T00:40:52.408 DEBUG:boto:Token: None 2023-09-30T00:40:52.408 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj3 2023-09-30T00:40:52.408 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:CVxq+r3s1Mxs8MiOrRNKPFfG2mc= 2023-09-30T00:40:52.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CVxq+r3s1Mxs8MiOrRNKPFfG2mc=', 'Content-Length': '0'} 2023-09-30T00:40:52.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034edf68711f1119c-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.410 DEBUG:boto:path=/dmuhsu-88/obj3 2023-09-30T00:40:52.410 DEBUG:boto:auth_path=/dmuhsu-88/obj3 2023-09-30T00:40:52.410 DEBUG:boto:Method: GET 2023-09-30T00:40:52.411 DEBUG:boto:Path: /dmuhsu-88/obj3 2023-09-30T00:40:52.411 DEBUG:boto:Data: 2023-09-30T00:40:52.411 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.411 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.411 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.411 DEBUG:boto:Params: {} 2023-09-30T00:40:52.411 DEBUG:boto:Token: None 2023-09-30T00:40:52.411 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj3 2023-09-30T00:40:52.411 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vFV3TOc+PlImxp99ZDDsXaRrUAQ= 2023-09-30T00:40:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vFV3TOc+PlImxp99ZDDsXaRrUAQ=', 'Content-Length': '0'} 2023-09-30T00:40:52.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000575a3ecf6e3bb6ad-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.413 DEBUG:boto:path=/dmuhsu-88/obj4 2023-09-30T00:40:52.413 DEBUG:boto:auth_path=/dmuhsu-88/obj4 2023-09-30T00:40:52.413 DEBUG:boto:Method: HEAD 2023-09-30T00:40:52.414 DEBUG:boto:Path: /dmuhsu-88/obj4 2023-09-30T00:40:52.414 DEBUG:boto:Data: 2023-09-30T00:40:52.414 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.414 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.414 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.414 DEBUG:boto:Params: {} 2023-09-30T00:40:52.414 DEBUG:boto:Token: None 2023-09-30T00:40:52.414 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj4 2023-09-30T00:40:52.415 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:c5S+EoqQVCkDR3F1G04NneZ9ETk= 2023-09-30T00:40:52.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c5S+EoqQVCkDR3F1G04NneZ9ETk=', 'Content-Length': '0'} 2023-09-30T00:40:52.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006552e27ab3dbd715-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.416 DEBUG:boto:path=/dmuhsu-88/obj4 2023-09-30T00:40:52.416 DEBUG:boto:auth_path=/dmuhsu-88/obj4 2023-09-30T00:40:52.416 DEBUG:boto:Method: GET 2023-09-30T00:40:52.417 DEBUG:boto:Path: /dmuhsu-88/obj4 2023-09-30T00:40:52.417 DEBUG:boto:Data: 2023-09-30T00:40:52.417 DEBUG:boto:Headers: {} 2023-09-30T00:40:52.417 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:40:52.417 DEBUG:boto:Port: 8000 2023-09-30T00:40:52.417 DEBUG:boto:Params: {} 2023-09-30T00:40:52.417 DEBUG:boto:Token: None 2023-09-30T00:40:52.417 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:40:52 GMT /dmuhsu-88/obj4 2023-09-30T00:40:52.417 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:yU7+8Ugtzp38tQktCoz/rZsAeeM= 2023-09-30T00:40:52.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yU7+8Ugtzp38tQktCoz/rZsAeeM=', 'Content-Length': '0'} 2023-09-30T00:40:52.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008262c07439094ed0-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:40:52.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.420 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:52.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.545 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:40:52.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.546 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:52.593 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.594 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.644 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:40:52.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.645 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:52.695 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.695 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.756 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:40:52.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.757 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:52.806 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.806 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.878 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:40:52.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.879 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:52.927 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:52.987 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:40:52.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:52.987 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.032 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.033 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.081 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:40:53.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.082 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.200 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:40:53.200 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 19, 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.201 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.202 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:40:53.203 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.204 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:40:53.205 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:40:53.207 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:40:53.208 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:40:53.209 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:40:53.210 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:40:53.211 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.221 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:40:53.222 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-85", 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-86", 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.223 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.224 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "allowed" 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.225 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.226 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:53.227 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:40:53.228 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:40:53.229 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:40:53.230 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:40:53.231 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:40:53.232 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:40:53.233 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:40:53.234 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:40:53.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.235 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.278 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.279 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.334 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:40:53.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.335 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.431 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.578 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:40:53.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.579 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.656 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.656 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.799 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:40:53.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-87', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-87', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.800 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-87 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-87 --source-bucket=dmuhsu-87 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:53.870 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:53.978 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:40:53.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-88', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-88', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:53.978 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-88 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-87 --source-bucket=dmuhsu-88 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:54.038 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:54.039 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:54.150 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:40:54.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:54.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:54.150 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:54.282 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:40:54.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:54.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:40:54.283 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:40:54.328 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:54.329 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:54.392 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:40:54.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:54.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:54.393 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:54.438 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:54.438 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:54.481 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:40:54.499 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:40:54.259118Z" 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:40:54.500 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:40:54.501 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:40:54.501 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:40:54.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:54.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:54.502 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:54.553 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:54.554 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:54.588 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:40:54.587+0000 7f92927a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:40:54.589 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:40:54.588+0000 7f92927a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:40:54.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:54.603 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:54.603 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:54.604 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:54.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:40:54.607 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:54.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:54.609 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:54Z" 2023-09-30T00:40:54.610 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:54.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:40:54Z" } 2023-09-30T00:40:54.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:40:54.611 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:40:54.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:40:54.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:40:59.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:59.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:40:59.613 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:40:59.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:40:59.666 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:40:59.700 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:40:59.699+0000 7fdfb9c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:40:59.700 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:40:59.700+0000 7fdfb9c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:40:59.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:40:59.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:40:59.717 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:59.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:40:59.719 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:40:59.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:40:59Z" 2023-09-30T00:40:59.721 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:40:59.722 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:40:59Z" } 2023-09-30T00:40:59.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:40:59.722 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:40:59.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:40:59.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:04.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:04.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:04.725 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:04.777 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:04.777 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:04.811 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:04.810+0000 7f3a20212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:04.812 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:04.811+0000 7f3a20212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:04.826 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:41:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:41:04.826 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:04.831 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:41:04.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:04.833 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:41:04Z" 2023-09-30T00:41:04.833 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:41:04.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:04Z" } 2023-09-30T00:41:04.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:04.834 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:04.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:04.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:09.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:09.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:09.836 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:09.888 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:09.889 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:09.922 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:09.921+0000 7fe5f231f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:09.923 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:09.923+0000 7fe5f231f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:09.938 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:41:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:41:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:41:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:41:09.938 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:09.939 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:09.940 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:09.941 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:09.942 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:09.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:41:09.944 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:41:09.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:09.945 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:41:09Z" 2023-09-30T00:41:09.945 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:41:09.946 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:09Z" } 2023-09-30T00:41:09.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:09.946 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:09.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:09.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:14.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:14.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:14.948 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:15.000 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:15.001 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:15.034 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:15.033+0000 7f2e5de13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:15.035 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:15.034+0000 7f2e5de13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:15.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:15.051 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:15.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:15.053 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:15.054 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:41:15.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:41:15Z" 2023-09-30T00:41:15.056 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:41:15.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:15Z" } 2023-09-30T00:41:15.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:15.057 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:15.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:15.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:20.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:20.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:20.059 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:20.112 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:20.113 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:20.147 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:20.147+0000 7fe43bc0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:20.148 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:20.147+0000 7fe43bc0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:20.163 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:41:20.163 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:41:20.163 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:41:20.163 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:41:20.163 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:20.164 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:41:20.165 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:20.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:20.167 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:20.168 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:20.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:20.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:20.169 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:20.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:20.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:41:20Z" 2023-09-30T00:41:20.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:41:20.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:20Z" } 2023-09-30T00:41:20.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:20.172 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:20.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:20.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:25.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:25.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:25.173 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:25.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:25.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:25.260 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:25.259+0000 7fc3ca530880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:25.261 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:41:25.260+0000 7fc3ca530880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:25.277 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:41:25.277 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:41:25.277 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:41:25.277 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:41:25.277 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:25.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:25.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:25.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:41:25.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:41:25.283 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:41:25.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:41:25.284 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:25Z" } 2023-09-30T00:41:25.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:25.285 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:25.285 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:25.285 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:41:25.285 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:41:25.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:25.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:25.286 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:25.339 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:25.339 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:25.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:25.376+0000 7f704bc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:25.378 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:25.377+0000 7f704bc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:25.393 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.394 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:25.395 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:25.396 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:25.397 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:25.398 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:25.399 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:41:25.400 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:25.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:25Z" } 2023-09-30T00:41:25.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:25.402 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:25.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:25.403 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:30.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:30.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:30.404 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:30.457 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:30.458 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:30.500 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:30.500+0000 7f232f332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:30.501 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:30.501+0000 7f232f332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:30.526 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:30.527 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:30.528 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.529 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:30.530 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:30.531 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:30.532 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:30.533 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:30.533 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:30Z" 2023-09-30T00:41:30.533 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:30.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:30Z" } 2023-09-30T00:41:30.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:30.535 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:30.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:30.535 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:35.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:35.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:35.537 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:35.590 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:35.590 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:35.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:35.643+0000 7f30671dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:35.644 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:35.644+0000 7f30671dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:35.662 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:35.662 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:35.662 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:35.662 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:35.662 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:35.663 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:35.664 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:35.665 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:35.666 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:35.667 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:35.668 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:35.669 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:35.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:35.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:35.669 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:35.670 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.671 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:35.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:35.672 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:35.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:35.672 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:35Z" 2023-09-30T00:41:35.672 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:35.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:35Z" } 2023-09-30T00:41:35.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:35.674 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:35.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:35.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:40.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:40.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:40.676 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:40.728 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:40.728 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:40.771 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:40.771+0000 7f204d811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:40.772 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:40.772+0000 7f204d811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:40.788 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:40.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:40.790 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:40.791 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:40.793 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:40.794 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:40.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:40Z" 2023-09-30T00:41:40.796 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:40.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:40Z" } 2023-09-30T00:41:40.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:40.798 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:40.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:40.798 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:45.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:45.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:45.800 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:45.853 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:45.853 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:45.894 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:45.893+0000 7fc6b3f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:45.895 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:45.894+0000 7fc6b3f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:45.911 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:45.911 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:45.911 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:45.912 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:45.913 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:45.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:45.915 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:45.916 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:45Z" 2023-09-30T00:41:45.917 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:45.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:45Z" } 2023-09-30T00:41:45.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:45.919 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:45.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:45.919 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:50.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:50.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:50.920 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:50.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:50.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:51.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:51.009+0000 7f5664410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:51.010 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:51.010+0000 7f5664410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:51.036 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:51.036 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:51.036 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:51.037 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:51.038 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:51.039 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:51.040 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:51.041 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:51.042 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:51.042 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:51.042 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:51.042 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:51.042 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:51.043 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:51.044 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:51.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:51.044 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:51Z" 2023-09-30T00:41:51.044 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:51.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:51Z" } 2023-09-30T00:41:51.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:51.045 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:51.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:51.045 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:41:56.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:56.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:41:56.048 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:41:56.100 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:41:56.101 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:41:56.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:56.151+0000 7f8b903dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:56.152 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:41:56.152+0000 7f8b903dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:41:56.167 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:41:56.167 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:41:56.167 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:41:56.167 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:41:56.167 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:41:56.168 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:41:56.168 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:56.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.168 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:41:56.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:56.169 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:41:56.171 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:41:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:56.172 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:56.173 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.173 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:56.173 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:41:56.173 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:56.173 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:41:56.174 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:56.175 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:56.175 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:41:56.175 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:41:56.175 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:41:56.176 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:41:56.176 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:41:56.176 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:41:56.176 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:41:56.176 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:41:56.177 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:41:56.177 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:41:56.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:56.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:41:56.177 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:41:56.178 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:41:56Z" 2023-09-30T00:41:56.179 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:41:56.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:41:56Z" } 2023-09-30T00:41:56.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:41:56.181 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:56.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:41:56.182 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:01.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:01.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:01.183 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:01.235 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:01.236 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:01.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:01.275+0000 7ff709533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:01.276 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:01.276+0000 7ff709533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:01.293 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.294 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:01.295 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:01.296 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:01.297 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:01.298 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:01.299 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:01Z" 2023-09-30T00:42:01.300 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:01.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:01Z" } 2023-09-30T00:42:01.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:01.302 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:01.302 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:01.302 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:06.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:06.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:06.304 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:06.356 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:06.357 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:06.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:06.401+0000 7fabb6ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:06.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:06.403+0000 7fabb6ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:06.420 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:06.420 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:06.420 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:06.420 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:06.421 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:06.422 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:06.423 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:06.424 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:06.425 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:06.426 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:06.427 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:06.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:06.428 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:06Z" 2023-09-30T00:42:06.428 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:06.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:06Z" } 2023-09-30T00:42:06.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:06.429 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:06.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:06.429 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:11.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:11.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:11.431 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:11.483 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:11.484 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:11.528 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:11.527+0000 7f23b392e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:11.529 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:11.528+0000 7f23b392e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:11.547 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:11.547 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:11.547 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:11.548 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.549 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:11.550 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:11.551 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:11.552 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:11.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:11.554 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:11Z" 2023-09-30T00:42:11.554 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:11.555 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:11Z" } 2023-09-30T00:42:11.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:11.555 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:11.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:11.555 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:16.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:16.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:16.557 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:16.611 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:16.611 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:16.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:16.648+0000 7f887512e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:16.651 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:16.650+0000 7f887512e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:16.667 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.668 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:16.669 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:16.670 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:16.671 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:16.672 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:16.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:16Z" 2023-09-30T00:42:16.674 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:16.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:16Z" } 2023-09-30T00:42:16.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:16.675 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:16.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:16.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:21.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:21.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:21.677 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:21.730 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:21.730 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:21.766 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:21.765+0000 7f5fa9711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:21.767 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:21.766+0000 7f5fa9711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:21.784 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:21.785 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:21.786 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.787 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:21.788 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:21.789 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:21.790 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:21.791 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:21.791 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:21Z" 2023-09-30T00:42:21.791 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:21.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:21Z" } 2023-09-30T00:42:21.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:21.792 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:21.792 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:21.792 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:26.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:26.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:26.794 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:26.846 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:26.846 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:26.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:26.890+0000 7f4538531880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:26.892 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:26.891+0000 7f4538531880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:26.907 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:26.907 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:26.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:26.910 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:26.911 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:26.912 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:26.913 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:26.914 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:26.914 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:26.914 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:26Z" 2023-09-30T00:42:26.914 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:26.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:26Z" } 2023-09-30T00:42:26.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:26.915 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:26.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:26.915 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:31.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:31.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:31.917 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:31.969 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:31.970 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:32.005 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:32.005+0000 7ff0e4215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:32.006 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:32.006+0000 7ff0e4215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:32.023 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:32.024 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:32.025 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:32.026 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:32.027 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.028 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:32.029 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:32.029 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:32.029 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:32.029 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:32Z" 2023-09-30T00:42:32.029 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:32.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:32Z" } 2023-09-30T00:42:32.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:32.030 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:32.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:32.030 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:37.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:37.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:37.032 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:37.085 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:37.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:37.142+0000 7f1325daf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:37.143 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:37.143+0000 7f1325daf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:37.158 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:37.159 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:37.160 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:37.161 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:37.163 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:37.164 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:37.165 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:37Z" 2023-09-30T00:42:37.165 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:37.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:37Z" } 2023-09-30T00:42:37.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:37.166 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:37.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:37.166 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:42.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:42.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:42.167 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:42.220 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:42.220 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:42.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:42.257+0000 7f73ff721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:42.258 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:42.258+0000 7f73ff721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:42.282 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:42.282 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:42.282 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:42.283 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:42.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:42.286 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:42.287 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:42.288 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:42.289 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:42.289 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:42Z" 2023-09-30T00:42:42.289 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:42.290 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:42Z" } 2023-09-30T00:42:42.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:42.290 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:42.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:42.290 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:47.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:47.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:47.292 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:47.347 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:47.402 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:47.402+0000 7f5695867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:47.403 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:47.403+0000 7f5695867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:47.425 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:47.425 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.426 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:47.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:47.427 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:47.427 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:47.427 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:47.427 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:47.428 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:47.429 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:47.430 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:47.430 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:47.430 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:47.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:47.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:47.431 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:47.433 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:47.433 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:47.433 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:47.433 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:47.433 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:47.434 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:47.435 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:47.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.435 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:47.435 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:47.435 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:47.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:47.436 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:47Z" 2023-09-30T00:42:47.436 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:47.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:47Z" } 2023-09-30T00:42:47.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:47.437 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:47.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:47.438 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:52.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:52.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:52.439 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:52.490 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:52.491 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:52.525 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:52.525+0000 7f8da3536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:52.526 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:52.526+0000 7f8da3536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:52.546 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:52.547 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:52.548 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:52.549 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:52.550 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:52.551 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:52.551 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:52.551 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:52.551 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:52.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:42:52.552 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:52.554 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:52.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:52Z" 2023-09-30T00:42:52.556 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:52.557 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034322.186786_1106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:38:42.186786Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:52Z" } 2023-09-30T00:42:52.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:52.558 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:52.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:52.558 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:42:57.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:57.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:42:57.560 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:42:57.613 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:42:57.613 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:42:57.649 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:57.649+0000 7f06eb810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:57.650 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:42:57.650+0000 7f06eb810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:42:57.665 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:42:57.665 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:42:57.665 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:57.666 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:42:57.667 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:57.668 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.669 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034453.507756_1140.1", 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:57.670 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:40:53.507756Z", 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:42:57.671 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:42:57.672 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:42:57Z" 2023-09-30T00:42:57.673 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:42:57.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034453.507756_1140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:53.507756Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:42:57Z" } 2023-09-30T00:42:57.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034453.507756_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:53.507756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:42:57.675 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:57.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034453.507756_1140.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:42:57.675 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034453.507756_1140.1 2023-09-30T00:43:02.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:02.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:02.677 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:02.729 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:02.730 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:02.787+0000 7f0d3e1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:02.788 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:02.788+0000 7f0d3e1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:02.813 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:02.814 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:02.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:02.816 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:02.817 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:02.818 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:02.819 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:02.820 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:02.820 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:02.820 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:02Z" 2023-09-30T00:43:02.820 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:02.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:02Z" } 2023-09-30T00:43:02.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:02.821 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:02.821 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:02.821 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:43:02.822 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:43:02.822 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:43:02.822 DEBUG:boto:Method: HEAD 2023-09-30T00:43:02.822 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:43:02.822 DEBUG:boto:Data: 2023-09-30T00:43:02.822 DEBUG:boto:Headers: {} 2023-09-30T00:43:02.822 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.823 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.823 DEBUG:boto:Params: {} 2023-09-30T00:43:02.823 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:43:02.823 DEBUG:boto:Token: None 2023-09-30T00:43:02.823 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-87/ 2023-09-30T00:43:02.824 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks= 2023-09-30T00:43:02.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks=', 'Content-Length': '0'} 2023-09-30T00:43:02.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007020b373a549b524-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.829 DEBUG:boto:path=/dmuhsu-87/obj5 2023-09-30T00:43:02.829 DEBUG:boto:auth_path=/dmuhsu-87/obj5 2023-09-30T00:43:02.830 DEBUG:boto:Method: PUT 2023-09-30T00:43:02.830 DEBUG:boto:Path: /dmuhsu-87/obj5 2023-09-30T00:43:02.830 DEBUG:boto:Data: 2023-09-30T00:43:02.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:02.830 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.830 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.830 DEBUG:boto:Params: {} 2023-09-30T00:43:02.830 DEBUG:boto:Token: None 2023-09-30T00:43:02.830 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-87/obj5 2023-09-30T00:43:02.831 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:mRxQ8kvYAp46oez4A91az5iK08c= 2023-09-30T00:43:02.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mRxQ8kvYAp46oez4A91az5iK08c='} 2023-09-30T00:43:02.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe3908832bdb976a-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.841 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:43:02.841 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:43:02.841 DEBUG:boto:Method: HEAD 2023-09-30T00:43:02.841 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:43:02.841 DEBUG:boto:Data: 2023-09-30T00:43:02.841 DEBUG:boto:Headers: {} 2023-09-30T00:43:02.841 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.841 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.841 DEBUG:boto:Params: {} 2023-09-30T00:43:02.842 DEBUG:boto:Token: None 2023-09-30T00:43:02.842 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-87/ 2023-09-30T00:43:02.842 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks= 2023-09-30T00:43:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks=', 'Content-Length': '0'} 2023-09-30T00:43:02.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009694bcaf238e7d95-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.844 DEBUG:boto:path=/dmuhsu-87/obj6 2023-09-30T00:43:02.844 DEBUG:boto:auth_path=/dmuhsu-87/obj6 2023-09-30T00:43:02.845 DEBUG:boto:Method: PUT 2023-09-30T00:43:02.845 DEBUG:boto:Path: /dmuhsu-87/obj6 2023-09-30T00:43:02.845 DEBUG:boto:Data: 2023-09-30T00:43:02.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:02.845 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.845 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.845 DEBUG:boto:Params: {} 2023-09-30T00:43:02.845 DEBUG:boto:Token: None 2023-09-30T00:43:02.845 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-87/obj6 2023-09-30T00:43:02.846 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:FNQzx4jeLTg5C95g4AJyQO4m9Jw= 2023-09-30T00:43:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FNQzx4jeLTg5C95g4AJyQO4m9Jw='} 2023-09-30T00:43:02.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f65e10a059b6bebd-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.852 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:43:02.852 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:43:02.852 DEBUG:boto:Method: HEAD 2023-09-30T00:43:02.852 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:43:02.852 DEBUG:boto:Data: 2023-09-30T00:43:02.853 DEBUG:boto:Headers: {} 2023-09-30T00:43:02.853 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.853 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.853 DEBUG:boto:Params: {} 2023-09-30T00:43:02.853 DEBUG:boto:Token: None 2023-09-30T00:43:02.853 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-88/ 2023-09-30T00:43:02.853 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o= 2023-09-30T00:43:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o=', 'Content-Length': '0'} 2023-09-30T00:43:02.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfb2d1f86b25ad27-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.855 DEBUG:boto:path=/dmuhsu-88/obj7 2023-09-30T00:43:02.856 DEBUG:boto:auth_path=/dmuhsu-88/obj7 2023-09-30T00:43:02.856 DEBUG:boto:Method: PUT 2023-09-30T00:43:02.856 DEBUG:boto:Path: /dmuhsu-88/obj7 2023-09-30T00:43:02.856 DEBUG:boto:Data: 2023-09-30T00:43:02.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:02.856 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.856 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.856 DEBUG:boto:Params: {} 2023-09-30T00:43:02.856 DEBUG:boto:Token: None 2023-09-30T00:43:02.857 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-88/obj7 2023-09-30T00:43:02.857 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:oy40ckbj819Erg+Cc0JSh/p3dks= 2023-09-30T00:43:02.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oy40ckbj819Erg+Cc0JSh/p3dks='} 2023-09-30T00:43:02.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d87e54f7ba02e63e-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.899 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:43:02.899 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:43:02.899 DEBUG:boto:Method: HEAD 2023-09-30T00:43:02.899 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:43:02.899 DEBUG:boto:Data: 2023-09-30T00:43:02.899 DEBUG:boto:Headers: {} 2023-09-30T00:43:02.899 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.899 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.900 DEBUG:boto:Params: {} 2023-09-30T00:43:02.900 DEBUG:boto:Token: None 2023-09-30T00:43:02.900 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-88/ 2023-09-30T00:43:02.900 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o= 2023-09-30T00:43:02.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o=', 'Content-Length': '0'} 2023-09-30T00:43:02.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7ede8e1491690f-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.903 DEBUG:boto:path=/dmuhsu-88/obj8 2023-09-30T00:43:02.903 DEBUG:boto:auth_path=/dmuhsu-88/obj8 2023-09-30T00:43:02.903 DEBUG:boto:Method: PUT 2023-09-30T00:43:02.903 DEBUG:boto:Path: /dmuhsu-88/obj8 2023-09-30T00:43:02.903 DEBUG:boto:Data: 2023-09-30T00:43:02.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:02.903 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:02.903 DEBUG:boto:Port: 8000 2023-09-30T00:43:02.903 DEBUG:boto:Params: {} 2023-09-30T00:43:02.903 DEBUG:boto:Token: None 2023-09-30T00:43:02.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:02 GMT /dmuhsu-88/obj8 2023-09-30T00:43:02.904 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:sI2PYCa0xbxP/a53TECnSwflCW0= 2023-09-30T00:43:02.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sI2PYCa0xbxP/a53TECnSwflCW0='} 2023-09-30T00:43:02.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004595785819cb1f68-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:02.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:02.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:02.910 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:02.954 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:02.954 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:02.986 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:43:03.000 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:03.000 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.001 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:40:54.259118Z" 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.002 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:03.003 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:03.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:43:03.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.004 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:03.056 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:03.090 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:03.089+0000 7fb54b3a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:03.091 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:03.090+0000 7fb54b3a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:03.105 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:03.105 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.106 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:03.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.109 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.110 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:03.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:43:03.112 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:03.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:03Z" } 2023-09-30T00:43:03.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:03.113 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:03.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:03.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:43:03.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:43:03.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.114 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:03.160 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:03.160 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:03.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:03.193+0000 7fb5ef136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:03.194 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:03.194+0000 7fb5ef136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:03.217 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:03.217 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:03.218 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:03.219 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:03.220 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:03.221 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:03.222 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.223 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:43:03.224 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:03.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:03Z" } 2023-09-30T00:43:03.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:03.226 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:03.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:03.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:43:03.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.226 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:03.273 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:03.273 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.350 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:03.351 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:03.352 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:03.352 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:03.353 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:03.353 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:43:03.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:03.353 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:03.404 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:03.404 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:03.450 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:03.450 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:03.450 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:03.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.452 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:03.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:03.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:03.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:43:03.456 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:03.457 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013510", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:39:19.485460Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024647", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:39:19.452905Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018614", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:39:19.393348Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:03Z" } 2023-09-30T00:43:03.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:43:03.457 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:03.458 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:43:03.458 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:43:08.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:08.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:08.459 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:08.513 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:08.513 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:08.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:08.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:43:08.566 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:08.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:08.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:08Z" 2023-09-30T00:43:08.569 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:08.570 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013510", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:39:19.485460Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024647", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:39:19.452905Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018614", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:39:19.393348Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:08Z" } 2023-09-30T00:43:08.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:43:08.571 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:08.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:43:08.571 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:43:13.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:13.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:13.572 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:13.627 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:13.627 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:13.677 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:13.678 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:13.680 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:13.681 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:13.682 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:13.683 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:13.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:13.683 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:13Z" 2023-09-30T00:43:13.683 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:13.684 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013510", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:39:19.485460Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024647", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:39:19.452905Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018614", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:39:19.393348Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:13Z" } 2023-09-30T00:43:13.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:43:13.684 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:13.684 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:43:13.684 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:43:18.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.685 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:18.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:18.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:18.790 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:18.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:18.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:18.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:18.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:18Z" 2023-09-30T00:43:18.796 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:18.797 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:18Z" } 2023-09-30T00:43:18.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:18.798 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:18.798 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:18.798 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:43:18.798 DEBUG:boto:path=/dmuhsu-88/ 2023-09-30T00:43:18.798 DEBUG:boto:auth_path=/dmuhsu-88/ 2023-09-30T00:43:18.798 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.798 DEBUG:boto:Path: /dmuhsu-88/ 2023-09-30T00:43:18.799 DEBUG:boto:Data: 2023-09-30T00:43:18.799 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.799 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.799 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.799 DEBUG:boto:Params: {} 2023-09-30T00:43:18.799 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:43:18.799 DEBUG:boto:Token: None 2023-09-30T00:43:18.800 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-88/ 2023-09-30T00:43:18.800 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:5KPFrF4y5qlrBdxmkCLSlCcIehM= 2023-09-30T00:43:18.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5KPFrF4y5qlrBdxmkCLSlCcIehM=', 'Content-Length': '0'} 2023-09-30T00:43:18.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000843500add6ad4dcd-0065176f26-20f2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.805 DEBUG:boto:path=/dmuhsu-88/obj5 2023-09-30T00:43:18.805 DEBUG:boto:auth_path=/dmuhsu-88/obj5 2023-09-30T00:43:18.805 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.805 DEBUG:boto:Path: /dmuhsu-88/obj5 2023-09-30T00:43:18.805 DEBUG:boto:Data: 2023-09-30T00:43:18.805 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.805 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.805 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.805 DEBUG:boto:Params: {} 2023-09-30T00:43:18.806 DEBUG:boto:Token: None 2023-09-30T00:43:18.806 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-88/obj5 2023-09-30T00:43:18.806 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:E3CDrG0BZM7fCYuTynVGrlmTM0s= 2023-09-30T00:43:18.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E3CDrG0BZM7fCYuTynVGrlmTM0s=', 'Content-Length': '0'} 2023-09-30T00:43:18.807 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cabb8fbb94c7df23-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.808 DEBUG:boto:path=/dmuhsu-88/obj6 2023-09-30T00:43:18.808 DEBUG:boto:auth_path=/dmuhsu-88/obj6 2023-09-30T00:43:18.808 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.808 DEBUG:boto:Path: /dmuhsu-88/obj6 2023-09-30T00:43:18.808 DEBUG:boto:Data: 2023-09-30T00:43:18.808 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.808 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.808 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.808 DEBUG:boto:Params: {} 2023-09-30T00:43:18.809 DEBUG:boto:Token: None 2023-09-30T00:43:18.809 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-88/obj6 2023-09-30T00:43:18.809 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:nuFKUvu4OVFwgKNqgbXwzJI1MW8= 2023-09-30T00:43:18.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nuFKUvu4OVFwgKNqgbXwzJI1MW8=', 'Content-Length': '0'} 2023-09-30T00:43:18.810 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b11772b5b07ccbcd-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.810 DEBUG:boto:path=/dmuhsu-88/obj7 2023-09-30T00:43:18.810 DEBUG:boto:auth_path=/dmuhsu-88/obj7 2023-09-30T00:43:18.810 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.811 DEBUG:boto:Path: /dmuhsu-88/obj7 2023-09-30T00:43:18.811 DEBUG:boto:Data: 2023-09-30T00:43:18.811 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.811 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.811 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.811 DEBUG:boto:Params: {} 2023-09-30T00:43:18.811 DEBUG:boto:Token: None 2023-09-30T00:43:18.812 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-88/obj7 2023-09-30T00:43:18.812 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:M19fHSLexUY9WnTveSS2gP0qCTg= 2023-09-30T00:43:18.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M19fHSLexUY9WnTveSS2gP0qCTg=', 'Content-Length': '0'} 2023-09-30T00:43:18.813 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a695169c71ad56cc-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.813 DEBUG:boto:path=/dmuhsu-88/obj8 2023-09-30T00:43:18.813 DEBUG:boto:auth_path=/dmuhsu-88/obj8 2023-09-30T00:43:18.813 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.813 DEBUG:boto:Path: /dmuhsu-88/obj8 2023-09-30T00:43:18.813 DEBUG:boto:Data: 2023-09-30T00:43:18.814 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.814 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.814 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.814 DEBUG:boto:Params: {} 2023-09-30T00:43:18.814 DEBUG:boto:Token: None 2023-09-30T00:43:18.814 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-88/obj8 2023-09-30T00:43:18.814 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:XCN0AIeaubpPZ61x0Wu97JeNVVU= 2023-09-30T00:43:18.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XCN0AIeaubpPZ61x0Wu97JeNVVU=', 'Content-Length': '0'} 2023-09-30T00:43:18.816 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0716a1fca1a738c-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.816 DEBUG:boto:path=/dmuhsu-87/ 2023-09-30T00:43:18.816 DEBUG:boto:auth_path=/dmuhsu-87/ 2023-09-30T00:43:18.816 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.816 DEBUG:boto:Path: /dmuhsu-87/ 2023-09-30T00:43:18.816 DEBUG:boto:Data: 2023-09-30T00:43:18.816 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.817 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.817 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.817 DEBUG:boto:Params: {} 2023-09-30T00:43:18.817 DEBUG:boto:Token: None 2023-09-30T00:43:18.817 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-87/ 2023-09-30T00:43:18.817 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:d0bjODNdvQHPDbBl20YNQxKY/bY= 2023-09-30T00:43:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d0bjODNdvQHPDbBl20YNQxKY/bY=', 'Content-Length': '0'} 2023-09-30T00:43:18.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089ab83798fdd8660-0065176f26-20f2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.819 DEBUG:boto:path=/dmuhsu-87/obj7 2023-09-30T00:43:18.819 DEBUG:boto:auth_path=/dmuhsu-87/obj7 2023-09-30T00:43:18.819 DEBUG:boto:Method: HEAD 2023-09-30T00:43:18.819 DEBUG:boto:Path: /dmuhsu-87/obj7 2023-09-30T00:43:18.820 DEBUG:boto:Data: 2023-09-30T00:43:18.820 DEBUG:boto:Headers: {} 2023-09-30T00:43:18.820 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:43:18.820 DEBUG:boto:Port: 8000 2023-09-30T00:43:18.820 DEBUG:boto:Params: {} 2023-09-30T00:43:18.820 DEBUG:boto:Token: None 2023-09-30T00:43:18.820 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:18 GMT /dmuhsu-87/obj7 2023-09-30T00:43:18.820 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ounF77NhOPISdLiMcgkUqw9HNOI= 2023-09-30T00:43:18.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ounF77NhOPISdLiMcgkUqw9HNOI=', 'Content-Length': '0'} 2023-09-30T00:43:18.821 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e5489a1feddeaa92-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:18.825 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-09-30T00:43:18.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.826 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:18.878 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:18.878 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:18.915 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:18.934 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:43:18.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:40:54.259118Z" 2023-09-30T00:43:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:18.938 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:43:18.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:18.939 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:18.940 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:18.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:43:18.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:18.941 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:18.988 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:18.988 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.018 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:19.018+0000 7f758cd21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.019 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:19.019+0000 7f758cd21880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:19.034 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.035 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.037 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.040 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:19Z" 2023-09-30T00:43:19.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:19.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:19Z" } 2023-09-30T00:43:19.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:19.042 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.042 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:43:19.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:43:19.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.043 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:19.089 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:19.090 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.133 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:19.132+0000 7fd960b29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.134 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:19.133+0000 7fd960b29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.150 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:19.151 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.152 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:19.153 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.154 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:19.155 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:19Z" 2023-09-30T00:43:19.157 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:19.158 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034454.259118_1179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:40:54.259118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:19Z" } 2023-09-30T00:43:19.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:19.158 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.158 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:43:19.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:19.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:19.159 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:19.200 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:19.201 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.265 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:19.266 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-89 2023-09-30T00:43:19.266 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:43:19.266 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:43:19.266 DEBUG:boto:Method: PUT 2023-09-30T00:43:19.266 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:43:19.266 DEBUG:boto:Data: 2023-09-30T00:43:19.267 DEBUG:boto:Headers: {} 2023-09-30T00:43:19.267 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:19.267 DEBUG:boto:Port: 8000 2023-09-30T00:43:19.267 DEBUG:boto:Params: {} 2023-09-30T00:43:19.267 DEBUG:boto:Token: None 2023-09-30T00:43:19.267 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:43:19 GMT /dmuhsu-89/ 2023-09-30T00:43:19.267 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:/PVqIQINENBGUxqAKTlhF/oGfcI= 2023-09-30T00:43:19.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/PVqIQINENBGUxqAKTlhF/oGfcI=', 'Content-Length': '0'} 2023-09-30T00:43:19.289 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8a2fb1f25a02e20-0065176f27-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:19.289 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=dmuhsu-90 2023-09-30T00:43:19.289 DEBUG:boto:path=/dmuhsu-90/ 2023-09-30T00:43:19.289 DEBUG:boto:auth_path=/dmuhsu-90/ 2023-09-30T00:43:19.289 DEBUG:boto:Method: PUT 2023-09-30T00:43:19.289 DEBUG:boto:Path: /dmuhsu-90/ 2023-09-30T00:43:19.289 DEBUG:boto:Data: 2023-09-30T00:43:19.289 DEBUG:boto:Headers: {} 2023-09-30T00:43:19.290 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:19.290 DEBUG:boto:Port: 8000 2023-09-30T00:43:19.290 DEBUG:boto:Params: {} 2023-09-30T00:43:19.290 DEBUG:boto:Token: None 2023-09-30T00:43:19.290 DEBUG:boto:StringToSign: PUT Sat, 30 Sep 2023 00:43:19 GMT /dmuhsu-90/ 2023-09-30T00:43:19.290 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:LwVJfJkBY+yOTag1b/6gk0Qj99M= 2023-09-30T00:43:19.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LwVJfJkBY+yOTag1b/6gk0Qj99M=', 'Content-Length': '0'} 2023-09-30T00:43:19.307 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003bf29630f6fb3df3-0065176f27-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:19.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.308 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:19.353 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.390 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:43:19.405 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:43:19.406 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:19.304980Z" 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.407 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:19.408 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:43:19.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.409 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:19.458 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:19.459 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.492 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:19.492+0000 7fb159b26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.493 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:19.492+0000 7fb159b26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.507 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:19.508 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:19.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.512 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:19.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:43:19.513 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:43:19.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:19.513 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:19Z" 2023-09-30T00:43:19.513 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:19.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:19Z" } 2023-09-30T00:43:19.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:19.514 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.514 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:43:19.515 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:43:19.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:19.515 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:19.560 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:19.560 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:19.600 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:19.599+0000 7f8f79024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.601 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:19.600+0000 7f8f79024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:19.619 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:19.619 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:19.619 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:19.619 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:19.619 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:19.620 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.621 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:19.622 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034599.280068_1180.1", 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:43:19.280068Z", 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:19.623 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:19.624 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:19Z" 2023-09-30T00:43:19.625 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:19.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.280068_1180.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.280068Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:19Z" } 2023-09-30T00:43:19.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.280068_1180.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.280068Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:19.626 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.280068_1180.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:19.626 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034599.304980_1187.1 target=1_1696034599.280068_1180.1 2023-09-30T00:43:24.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:24.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:24.628 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:24.684 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:24.685 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:24.724 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:24.723+0000 7f8cfa7dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:24.725 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:24.724+0000 7f8cfa7dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:24.739 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:24.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.741 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:24.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:24.743 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:24.744 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:24.745 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:24Z" 2023-09-30T00:43:24.746 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:24.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:24Z" } 2023-09-30T00:43:24.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:24.747 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:24.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:24.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:43:24.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.748 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:24.794 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:24.795 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:24.844 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:24.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.845 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:24.893 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:24.893 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:24.965 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:24.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-89', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-89', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:24.966 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-89 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-89 --dest-bucket=dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.015 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.076 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:25.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.077 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-90 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-89 --dest-bucket=dmuhsu-90 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.127 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.177 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:25.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-89', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.178 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-89 --dest-bucket=dmuhsu-90 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.228 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.291 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-89","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:43:25.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.292 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.338 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.338 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.397 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-89","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:43:25.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.398 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.442 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.442 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.494 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:25.494 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:25.494 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 20, 2023-09-30T00:43:25.494 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:43:25.495 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:43:25.496 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:43:25.497 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:43:25.498 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.499 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:43:25.500 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:43:25.501 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:43:25.502 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:43:25.503 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.504 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.505 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.506 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:43:25.507 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:43:25.508 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-90", 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.509 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.510 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.511 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-89", 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:43:25.512 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.514 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-90", 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:43:25.515 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-90", 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:43:25.516 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:43:25.518 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "enabled" 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.519 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:43:25.520 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:43:25.522 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:43:25.523 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:43:25.524 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.525 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:43:25.526 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:43:25.526 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:43:25.526 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:43:25.526 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:43:25.526 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:43:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:43:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:43:25.527 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:25.528 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:25.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:43:25.528 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:43:25.574 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:25.574 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:25.668 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-89","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:43:25.669 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:43:25.669 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:43:25.669 DEBUG:boto:Method: HEAD 2023-09-30T00:43:25.669 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:43:25.669 DEBUG:boto:Data: 2023-09-30T00:43:25.669 DEBUG:boto:Headers: {} 2023-09-30T00:43:25.669 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:25.670 DEBUG:boto:Port: 8000 2023-09-30T00:43:25.670 DEBUG:boto:Params: {} 2023-09-30T00:43:25.670 DEBUG:boto:Token: None 2023-09-30T00:43:25.670 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:25 GMT /dmuhsu-89/ 2023-09-30T00:43:25.670 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AqjQ80MM6vq2R8uE8ikJdDJ2r3A= 2023-09-30T00:43:25.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AqjQ80MM6vq2R8uE8ikJdDJ2r3A=', 'Content-Length': '0'} 2023-09-30T00:43:25.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004629c12a25eba2b4-0065176f2d-28a4-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:25.918 DEBUG:boto:path=/dmuhsu-89/obj1 2023-09-30T00:43:25.918 DEBUG:boto:auth_path=/dmuhsu-89/obj1 2023-09-30T00:43:25.918 DEBUG:boto:Method: PUT 2023-09-30T00:43:25.918 DEBUG:boto:Path: /dmuhsu-89/obj1 2023-09-30T00:43:25.918 DEBUG:boto:Data: 2023-09-30T00:43:25.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:25.919 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:25.919 DEBUG:boto:Port: 8000 2023-09-30T00:43:25.919 DEBUG:boto:Params: {} 2023-09-30T00:43:25.919 DEBUG:boto:Token: None 2023-09-30T00:43:25.919 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:25 GMT /dmuhsu-89/obj1 2023-09-30T00:43:25.919 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:ydQJ56MzGC3nm8Z1KCXnkbp4lgs= 2023-09-30T00:43:25.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ydQJ56MzGC3nm8Z1KCXnkbp4lgs='} 2023-09-30T00:43:25.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b05dc22475758e25-0065176f2d-28a4-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:25.948 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:43:25.948 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:43:25.948 DEBUG:boto:Method: HEAD 2023-09-30T00:43:25.948 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:43:25.948 DEBUG:boto:Data: 2023-09-30T00:43:25.948 DEBUG:boto:Headers: {} 2023-09-30T00:43:25.948 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:25.949 DEBUG:boto:Port: 8000 2023-09-30T00:43:25.949 DEBUG:boto:Params: {} 2023-09-30T00:43:25.949 DEBUG:boto:Token: None 2023-09-30T00:43:25.949 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:43:25 GMT /dmuhsu-89/ 2023-09-30T00:43:25.949 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:AqjQ80MM6vq2R8uE8ikJdDJ2r3A= 2023-09-30T00:43:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AqjQ80MM6vq2R8uE8ikJdDJ2r3A=', 'Content-Length': '0'} 2023-09-30T00:43:25.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045624d135bdc4e41-0065176f2d-28a4-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:25.955 DEBUG:boto:path=/dmuhsu-89/obj2 2023-09-30T00:43:25.955 DEBUG:boto:auth_path=/dmuhsu-89/obj2 2023-09-30T00:43:25.955 DEBUG:boto:Method: PUT 2023-09-30T00:43:25.955 DEBUG:boto:Path: /dmuhsu-89/obj2 2023-09-30T00:43:25.955 DEBUG:boto:Data: 2023-09-30T00:43:25.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:43:25.956 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:43:25.956 DEBUG:boto:Port: 8000 2023-09-30T00:43:25.956 DEBUG:boto:Params: {} 2023-09-30T00:43:25.956 DEBUG:boto:Token: None 2023-09-30T00:43:25.956 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:43:25 GMT /dmuhsu-89/obj2 2023-09-30T00:43:25.956 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DlTlxfotxo1VezTSfwWxncMBM+Y= 2023-09-30T00:43:25.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DlTlxfotxo1VezTSfwWxncMBM+Y='} 2023-09-30T00:43:25.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f50e8d75489e627-0065176f2d-28a4-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:43:25.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:25.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:25.999 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:26.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:26.064 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:26.150 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:43:26.195 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:43:26.196 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:19.304980Z" 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.197 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:26.198 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:26.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:43:26.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.199 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:26.246 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:26.246 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:26.276 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:26.275+0000 7f6a9a51e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:26.276 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:43:26.276+0000 7f6a9a51e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:26Z" 2023-09-30T00:43:26.297 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:26.297 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:26Z" } 2023-09-30T00:43:26.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:26.298 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:26.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:26.298 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:43:26.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:43:26.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.299 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:26.358 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:26.431 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:26.430+0000 7f40aa9a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:26.432 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:43:26.431+0000 7f40aa9a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:43:26.479 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:26.480 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:26.481 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:43:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.484 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:43:26Z" 2023-09-30T00:43:26.485 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:43:26.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:43:26Z" } 2023-09-30T00:43:26.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:43:26.487 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:26.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:43:26.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:43:26.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.487 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:26.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:26.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.647 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:43:26.649 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:43:26.649 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:26.650 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:43:26.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:26.650 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:26.698 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:26.698 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:26.744 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:26.744 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:26Z" 2023-09-30T00:43:26.750 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:26.751 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:26Z" } 2023-09-30T00:43:26.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:26.751 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:26.751 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:26.752 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:31.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:31.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:31.753 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:31.807 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:31.807 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:31.859 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:31.859 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.860 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:31.861 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:31.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:31.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:31.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:31.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:31.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:31.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:31.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:31.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:31.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:31.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:31.868 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:31.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.869 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:31.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:31.869 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:31.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:31.870 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:31Z" 2023-09-30T00:43:31.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:31.871 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:31Z" } 2023-09-30T00:43:31.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:31.871 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:31.872 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:31.872 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:36.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:36.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:36.873 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:36.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:36.928 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:36.981 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:36.982 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:36.983 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:36.984 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:36Z" 2023-09-30T00:43:36.987 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:36.988 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:36Z" } 2023-09-30T00:43:36.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:36.988 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:36.988 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:36.988 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:41.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:41.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:41.990 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:42.044 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:42.045 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:42.095 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:42.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:42.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:42.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:42.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:42.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:42.102 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:42Z" 2023-09-30T00:43:42.102 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:42.102 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:42Z" } 2023-09-30T00:43:42.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:42.103 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:42.103 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:42.103 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:47.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:47.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:47.105 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:47.159 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:47.159 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:47.210 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:47.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:47.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:47.213 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:47.214 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:47.215 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:47.216 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:47.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:47.216 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:47Z" 2023-09-30T00:43:47.216 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:47.217 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:47Z" } 2023-09-30T00:43:47.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:47.217 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:47.217 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:47.217 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:52.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:52.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:52.219 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:52.273 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:52.273 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:52.325 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:52.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.327 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:52.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:52.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:52.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:52.327 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:52Z" 2023-09-30T00:43:52.336 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:52.336 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:52Z" } 2023-09-30T00:43:52.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:52.337 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:52.337 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:52.337 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:43:57.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:57.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:43:57.339 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:43:57.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:43:57.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:43:57.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:57.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:57.446 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:43:57.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:43:57.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:43:57.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:43:57.450 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:43:57.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:43:57.450 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:43:57Z" 2023-09-30T00:43:57.450 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:43:57.451 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:43:57Z" } 2023-09-30T00:43:57.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:43:57.451 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:43:57.451 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:43:57.451 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:02.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:02.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:02.453 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:02.508 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:02.508 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:02.559 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:02.560 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:02.561 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:02.562 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:02.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:02.564 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:02.565 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:02Z" 2023-09-30T00:44:02.566 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:02.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:02Z" } 2023-09-30T00:44:02.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:02.567 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:02.567 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:02.567 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:07.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:07.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:07.569 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:07.625 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:07.625 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:07.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:07.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:07.678 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:07.679 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:07.680 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:07.681 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:07Z" 2023-09-30T00:44:07.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:07.683 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:07Z" } 2023-09-30T00:44:07.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:07.683 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:07.683 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:07.683 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:12.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:12.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:12.684 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:12.738 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:12.739 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:12.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:12.790 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:12.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:12.792 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:12.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:12.794 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:12.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:12.795 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:12Z" 2023-09-30T00:44:12.795 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:12.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:12Z" } 2023-09-30T00:44:12.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:12.796 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:12.796 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:12.796 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:17.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:17.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:17.798 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:17.852 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:17.853 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:17.903 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:17.904 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:17.904 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:17.904 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:17.904 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:17.904 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:17.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:17.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:17Z" 2023-09-30T00:44:17.912 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:17.913 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:17Z" } 2023-09-30T00:44:17.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:17.913 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:17.914 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:17.914 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:22.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:22.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:22.917 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:22.973 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:22.973 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:23.023 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:23.024 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:23.026 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:23.027 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:23.028 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:23.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:23Z" 2023-09-30T00:44:23.030 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:23.031 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:23Z" } 2023-09-30T00:44:23.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:23.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:23.032 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:23.032 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:28.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:28.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:28.034 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:28.089 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:28.089 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:28.139 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:28Z" 2023-09-30T00:44:28.145 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:28.146 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:28Z" } 2023-09-30T00:44:28.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:28.147 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:28.147 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:28.147 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:33.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:33.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:33.148 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:33.203 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:33.203 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:33.256 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:33.256 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:33.257 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:33.258 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:33.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:33.261 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:33.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:33.263 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:33Z" 2023-09-30T00:44:33.263 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:33.264 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:33Z" } 2023-09-30T00:44:33.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:33.264 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:33.264 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:33.264 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:38.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:38.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:38.265 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:38.320 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:38.320 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:38.370 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:38.370 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:38.370 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:38.371 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:38.372 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:38.373 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:38.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:38Z" 2023-09-30T00:44:38.376 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:38.377 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:38Z" } 2023-09-30T00:44:38.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:38.378 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:38.378 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:38.378 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:43.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:43.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:43.380 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:43.435 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:43.435 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:43.485 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:43.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:43.492 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:43.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:43.492 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:43Z" 2023-09-30T00:44:43.492 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:43.493 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:43Z" } 2023-09-30T00:44:43.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:43.493 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:43.493 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:43.493 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:48.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:48.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:48.495 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:48.550 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:48.550 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:48.601 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:48.608 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:48.608 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:48.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:48.608 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:48Z" 2023-09-30T00:44:48.608 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:48.609 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:48Z" } 2023-09-30T00:44:48.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:48.609 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:48.610 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:48.610 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:53.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:53.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:53.612 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:53.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:53.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:53.718 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:53.718 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:53.719 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:53.720 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:53.721 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:53.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:44:53.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:53Z" 2023-09-30T00:44:53.724 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:53.725 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024927", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:02.863822Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:02.851293Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:53Z" } 2023-09-30T00:44:53.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:44:53.726 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:53.726 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:44:53.726 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000025067 target=00000000000000000000:00000000000000024927, shard=3 master=00000000000000000000:00000000000000018894 target=00000000000000000000:00000000000000018754 2023-09-30T00:44:58.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:58.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:44:58.728 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:44:58.787 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:58.787 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.834 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:58.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:58.836 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:58.837 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:44:58.838 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:58.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:44:58Z" 2023-09-30T00:44:58.840 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:44:58.841 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:44:58Z" } 2023-09-30T00:44:58.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:44:58.842 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:58.842 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:44:58.842 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:44:58.842 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:44:58.843 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:44:58.843 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.843 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:44:58.843 DEBUG:boto:Data: 2023-09-30T00:44:58.843 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.843 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.843 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.843 DEBUG:boto:Params: {} 2023-09-30T00:44:58.844 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:44:58.844 DEBUG:boto:Token: None 2023-09-30T00:44:58.844 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-89/ 2023-09-30T00:44:58.845 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:2XX4jjkzE26UP4caXG9sNq/2uuA= 2023-09-30T00:44:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2XX4jjkzE26UP4caXG9sNq/2uuA=', 'Content-Length': '0'} 2023-09-30T00:44:58.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f010e54e0a3cfaed-0065176f8a-2166-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.851 DEBUG:boto:path=/dmuhsu-89/obj1 2023-09-30T00:44:58.851 DEBUG:boto:auth_path=/dmuhsu-89/obj1 2023-09-30T00:44:58.851 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.851 DEBUG:boto:Path: /dmuhsu-89/obj1 2023-09-30T00:44:58.851 DEBUG:boto:Data: 2023-09-30T00:44:58.851 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.851 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.852 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.852 DEBUG:boto:Params: {} 2023-09-30T00:44:58.852 DEBUG:boto:Token: None 2023-09-30T00:44:58.852 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-89/obj1 2023-09-30T00:44:58.852 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:a8C7XRtUqECkUK8C6TQLSpA5SJ4= 2023-09-30T00:44:58.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a8C7XRtUqECkUK8C6TQLSpA5SJ4=', 'Content-Length': '0'} 2023-09-30T00:44:58.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1bf6c5d35cb9cfb-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.855 DEBUG:boto:path=/dmuhsu-89/obj1 2023-09-30T00:44:58.855 DEBUG:boto:auth_path=/dmuhsu-89/obj1 2023-09-30T00:44:58.855 DEBUG:boto:Method: GET 2023-09-30T00:44:58.855 DEBUG:boto:Path: /dmuhsu-89/obj1 2023-09-30T00:44:58.855 DEBUG:boto:Data: 2023-09-30T00:44:58.855 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.855 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.855 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.855 DEBUG:boto:Params: {} 2023-09-30T00:44:58.856 DEBUG:boto:Token: None 2023-09-30T00:44:58.856 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-89/obj1 2023-09-30T00:44:58.856 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:+KDEwosDG/OMTINk+da5DhRItI4= 2023-09-30T00:44:58.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+KDEwosDG/OMTINk+da5DhRItI4=', 'Content-Length': '0'} 2023-09-30T00:44:58.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b924f404ed8fe38b-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.858 DEBUG:boto:path=/dmuhsu-89/obj2 2023-09-30T00:44:58.858 DEBUG:boto:auth_path=/dmuhsu-89/obj2 2023-09-30T00:44:58.858 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.859 DEBUG:boto:Path: /dmuhsu-89/obj2 2023-09-30T00:44:58.859 DEBUG:boto:Data: 2023-09-30T00:44:58.859 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.859 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.859 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.859 DEBUG:boto:Params: {} 2023-09-30T00:44:58.860 DEBUG:boto:Token: None 2023-09-30T00:44:58.860 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-89/obj2 2023-09-30T00:44:58.860 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:DiWid3j6YoJ4fKFLck2VwyEqlUg= 2023-09-30T00:44:58.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DiWid3j6YoJ4fKFLck2VwyEqlUg=', 'Content-Length': '0'} 2023-09-30T00:44:58.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe18f688b9633560-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.862 DEBUG:boto:path=/dmuhsu-89/obj2 2023-09-30T00:44:58.862 DEBUG:boto:auth_path=/dmuhsu-89/obj2 2023-09-30T00:44:58.862 DEBUG:boto:Method: GET 2023-09-30T00:44:58.862 DEBUG:boto:Path: /dmuhsu-89/obj2 2023-09-30T00:44:58.862 DEBUG:boto:Data: 2023-09-30T00:44:58.862 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.862 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.862 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.863 DEBUG:boto:Params: {} 2023-09-30T00:44:58.863 DEBUG:boto:Token: None 2023-09-30T00:44:58.863 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-89/obj2 2023-09-30T00:44:58.863 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:QG4ADfZbarZQDlP9mdqUncO/UOQ= 2023-09-30T00:44:58.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QG4ADfZbarZQDlP9mdqUncO/UOQ=', 'Content-Length': '0'} 2023-09-30T00:44:58.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c86f5c32ec82f0d-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.865 DEBUG:boto:path=/dmuhsu-90/ 2023-09-30T00:44:58.865 DEBUG:boto:auth_path=/dmuhsu-90/ 2023-09-30T00:44:58.865 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.866 DEBUG:boto:Path: /dmuhsu-90/ 2023-09-30T00:44:58.866 DEBUG:boto:Data: 2023-09-30T00:44:58.866 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.866 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.866 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.866 DEBUG:boto:Params: {} 2023-09-30T00:44:58.866 DEBUG:boto:Token: None 2023-09-30T00:44:58.866 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-90/ 2023-09-30T00:44:58.867 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:0iL2sVYuaA2oHvjv1FJuCvWRaxU= 2023-09-30T00:44:58.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0iL2sVYuaA2oHvjv1FJuCvWRaxU=', 'Content-Length': '0'} 2023-09-30T00:44:58.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2d04df9281fa92b-0065176f8a-2166-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.869 DEBUG:boto:path=/dmuhsu-90/obj1 2023-09-30T00:44:58.869 DEBUG:boto:auth_path=/dmuhsu-90/obj1 2023-09-30T00:44:58.869 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.869 DEBUG:boto:Path: /dmuhsu-90/obj1 2023-09-30T00:44:58.870 DEBUG:boto:Data: 2023-09-30T00:44:58.870 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.870 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.870 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.870 DEBUG:boto:Params: {} 2023-09-30T00:44:58.870 DEBUG:boto:Token: None 2023-09-30T00:44:58.870 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-90/obj1 2023-09-30T00:44:58.870 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:v9d34BBY3qlOTKeyPW6NQMD0y9M= 2023-09-30T00:44:58.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:v9d34BBY3qlOTKeyPW6NQMD0y9M=', 'Content-Length': '0'} 2023-09-30T00:44:58.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a10adda39c916e23-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.872 DEBUG:boto:path=/dmuhsu-90/obj1 2023-09-30T00:44:58.872 DEBUG:boto:auth_path=/dmuhsu-90/obj1 2023-09-30T00:44:58.872 DEBUG:boto:Method: GET 2023-09-30T00:44:58.873 DEBUG:boto:Path: /dmuhsu-90/obj1 2023-09-30T00:44:58.873 DEBUG:boto:Data: 2023-09-30T00:44:58.873 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.873 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.873 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.873 DEBUG:boto:Params: {} 2023-09-30T00:44:58.873 DEBUG:boto:Token: None 2023-09-30T00:44:58.873 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-90/obj1 2023-09-30T00:44:58.873 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:X+/UKIOsd03+4PiwODVUAFLp8Fk= 2023-09-30T00:44:58.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X+/UKIOsd03+4PiwODVUAFLp8Fk=', 'Content-Length': '0'} 2023-09-30T00:44:58.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bac1b37f1afe0ff-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.875 DEBUG:boto:path=/dmuhsu-90/obj2 2023-09-30T00:44:58.875 DEBUG:boto:auth_path=/dmuhsu-90/obj2 2023-09-30T00:44:58.876 DEBUG:boto:Method: HEAD 2023-09-30T00:44:58.876 DEBUG:boto:Path: /dmuhsu-90/obj2 2023-09-30T00:44:58.876 DEBUG:boto:Data: 2023-09-30T00:44:58.876 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.876 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.876 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.876 DEBUG:boto:Params: {} 2023-09-30T00:44:58.876 DEBUG:boto:Token: None 2023-09-30T00:44:58.876 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-90/obj2 2023-09-30T00:44:58.877 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:fheZucJMQxV1MTWalbB49pflWkM= 2023-09-30T00:44:58.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:fheZucJMQxV1MTWalbB49pflWkM=', 'Content-Length': '0'} 2023-09-30T00:44:58.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd4ba6458df703a8-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.878 DEBUG:boto:path=/dmuhsu-90/obj2 2023-09-30T00:44:58.878 DEBUG:boto:auth_path=/dmuhsu-90/obj2 2023-09-30T00:44:58.878 DEBUG:boto:Method: GET 2023-09-30T00:44:58.879 DEBUG:boto:Path: /dmuhsu-90/obj2 2023-09-30T00:44:58.879 DEBUG:boto:Data: 2023-09-30T00:44:58.879 DEBUG:boto:Headers: {} 2023-09-30T00:44:58.879 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:44:58.879 DEBUG:boto:Port: 8000 2023-09-30T00:44:58.879 DEBUG:boto:Params: {} 2023-09-30T00:44:58.879 DEBUG:boto:Token: None 2023-09-30T00:44:58.879 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:44:58 GMT /dmuhsu-90/obj2 2023-09-30T00:44:58.879 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:MVkgRUizFvbGHYdzBF96ZKb7Yxk= 2023-09-30T00:44:58.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:44:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MVkgRUizFvbGHYdzBF96ZKb7Yxk=', 'Content-Length': '0'} 2023-09-30T00:44:58.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:43:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000386ad6457a883dc1-0065176f8a-2166-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:44:58.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:58.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:58.882 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:58.936 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:58.936 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.009 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:44:59.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.010 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.059 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.059 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.092 INFO:teuthology.orchestra.run.smithi175.stderr:WARNING: cannot find source zone id for name=* 2023-09-30T00:44:59.092 INFO:teuthology.orchestra.run.smithi175.stderr:WARNING: cannot find source zone id for name=* 2023-09-30T00:44:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:44:59.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.111 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.156 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.206 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:44:59.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.207 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.251 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.252 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 21, 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2023-09-30T00:44:59.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": true, 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2023-09-30T00:44:59.308 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:44:59.309 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:44:59.310 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.311 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": true, 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:44:59.312 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": false, 2023-09-30T00:44:59.313 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": true, 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": false, 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": false, 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "", 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "supported_features": [ 2023-09-30T00:44:59.314 INFO:teuthology.orchestra.run.smithi175.stdout: "compress-encrypted", 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2023-09-30T00:44:59.315 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [ 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-group", 2023-09-30T00:44:59.316 INFO:teuthology.orchestra.run.smithi175.stdout: "data_flow": { 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "symmetrical": [ 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-flow1", 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.317 INFO:teuthology.orchestra.run.smithi175.stdout: "pipes": [ 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-89", 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:44:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.319 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe-dmuhsu-90", 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.320 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-89", 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "dmuhsu-90", 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:44:59.321 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:44:59.322 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "sync-pipe", 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: "*" 2023-09-30T00:44:59.323 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": { 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket": "*", 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "*" 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "params": { 2023-09-30T00:44:59.324 INFO:teuthology.orchestra.run.smithi175.stdout: "source": { 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: "filter": { 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [] 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: "dest": {}, 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: "priority": 0, 2023-09-30T00:44:59.325 INFO:teuthology.orchestra.run.smithi175.stdout: "mode": "system", 2023-09-30T00:44:59.326 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "" 2023-09-30T00:44:59.326 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "allowed" 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled_features": [ 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: "resharding" 2023-09-30T00:44:59.327 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2885950644 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2464975307 2023-09-30T00:44:59.328 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1018467004 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2023-09-30T00:44:59.329 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2023-09-30T00:44:59.330 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "user_ratelimit": { 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:44:59.331 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_ratelimit": { 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:44:59.332 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "anonymous_ratelimit": { 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_ops": 0, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_ops": 0, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "max_read_bytes": 0, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "max_write_bytes": 0, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:44:59.333 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2023-09-30T00:44:59.334 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:44:59.334 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:44:59.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.334 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.380 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.440 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:44:59.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.441 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.497 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.498 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.639 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:44:59.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.639 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.707 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.707 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:44:59.864 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:44:59.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-89', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-89', '--source-bucket=*', '--dest-bucket=dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-89', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-89', '--source-bucket=*', '--dest-bucket=dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:44:59.864 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-89 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-89 '--source-bucket=*' --dest-bucket=dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:44:59.931 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:44:59.932 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.094 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:45:00.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-89', '--source-bucket=*', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-89', '--source-bucket=*', '--dest-bucket=dmuhsu-90', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.095 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-90 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-89 '--source-bucket=*' --dest-bucket=dmuhsu-90 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:45:00.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:00.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.281 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:45:00.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.282 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:45:00.335 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:00.336 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.434 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"*","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:45:00.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:45:00.435 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:45:00.480 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:00.481 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.561 INFO:teuthology.orchestra.run.smithi175.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-89","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-90","source":{"bucket":"dmuhsu-89","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-90","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:45:00.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:00.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:00.562 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:00.606 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:00.606 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.649 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:45:00.664 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:00.665 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:45:00.395556Z" 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:00.666 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:45:00.667 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:00.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:45:00.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:00.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:00.668 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:00.720 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:00.720 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:00.752 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:00.751+0000 7f170b819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:00.753 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:00.752+0000 7f170b819880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:00.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:00.768 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:00.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:00.770 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:00.770 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:00.770 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:00.771 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:00.772 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:00Z" 2023-09-30T00:45:00.774 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:00.775 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:00Z" } 2023-09-30T00:45:00.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:00.776 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:00.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:00.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:05.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:05.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:05.777 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:05.832 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:05.833 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:05.866 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:05.865+0000 7feaa3529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:05.866 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:05.866+0000 7feaa3529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:05.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:05.881 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:05.881 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:05.881 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:05.881 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:05.882 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:05.883 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:05.884 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:05.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:05.886 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:05Z" 2023-09-30T00:45:05.887 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:05.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:05Z" } 2023-09-30T00:45:05.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:05.889 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:05.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:05.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:10.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:10.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:10.891 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:10.944 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:10.944 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:10.978 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:10.977+0000 7f121e536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:10.978 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:10.978+0000 7f121e536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:10.993 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:10.993 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:10.993 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:10.994 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:10.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:10.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:10.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:10.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:10.999 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:11.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:11.000 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:10Z" 2023-09-30T00:45:11.000 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:11.001 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:10Z" } 2023-09-30T00:45:11.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:11.001 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:11.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:11.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:16.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:16.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:16.003 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:16.055 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:16.055 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:16.089 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:16.089+0000 7f8b2d89d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:16.090 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:16.090+0000 7f8b2d89d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:16.105 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:16.106 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:16.107 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:16.108 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:16.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:16Z" 2023-09-30T00:45:16.111 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:16.112 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:16Z" } 2023-09-30T00:45:16.112 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:16.113 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:16.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:16.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:21.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:21.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:21.114 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:21.168 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:21.168 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:21.202 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:21.202+0000 7f733f123880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:21.203 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:21.203+0000 7f733f123880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:21.218 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:21.219 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:21.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:21.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:21.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:21Z" 2023-09-30T00:45:21.224 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:21.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:21Z" } 2023-09-30T00:45:21.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:21.226 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:21.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:21.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:26.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:26.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:26.227 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:26.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:26.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:26.314 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:26.313+0000 7f2a84d11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:26.315 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:26.314+0000 7f2a84d11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:26.329 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034599.304980_1187.1", 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:19.304980Z", 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:26Z" 2023-09-30T00:45:26.336 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:26.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034599.304980_1187.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:19.304980Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:26Z" } 2023-09-30T00:45:26.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034599.304980_1187.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:19.304980Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:26.337 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:26.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034599.304980_1187.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:26.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034599.304980_1187.1 2023-09-30T00:45:31.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:31.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:31.340 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:31.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:31.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:31.428 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:31.427+0000 7f1535b2f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:31.429 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:31.428+0000 7f1535b2f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:31.443 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:45:00.395556Z", 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:31Z" 2023-09-30T00:45:31.451 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:31.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034700.395556_1259.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:45:00.395556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:31Z" } 2023-09-30T00:45:31.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034700.395556_1259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:45:00.395556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:31.453 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:31.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:31.453 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:45:31.453 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:45:31.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:31.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:31.454 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:31.506 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:31.506 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:31.551 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:31.551+0000 7f03cab22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:31.552 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:31.552+0000 7f03cab22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:31.569 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:45:31.569 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:45:31.569 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:45:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:31.571 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:31.572 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.573 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034699.581181_1227.1", 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:44:59.581181Z", 2023-09-30T00:45:31.574 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:31.575 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:45:31.576 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:45:31Z" 2023-09-30T00:45:31.577 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:45:31.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034699.581181_1227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:44:59.581181Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:31Z" } 2023-09-30T00:45:31.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034699.581181_1227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:44:59.581181Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:31.579 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:31.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034699.581181_1227.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:31.579 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1696034700.395556_1259.1 target=1_1696034699.581181_1227.1 2023-09-30T00:45:36.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.580 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:36.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:36.633 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:36.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:36.673+0000 7fef6bd2d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:36.674 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:36.674+0000 7fef6bd2d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:36.689 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:36.690 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:36.691 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:36.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.693 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:45:00.395556Z", 2023-09-30T00:45:36.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:36.695 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.696 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:45:36.697 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:45:36.697 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:45:36.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.697 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:45:36Z" 2023-09-30T00:45:36.697 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:45:36.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034700.395556_1259.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:45:00.395556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:36Z" } 2023-09-30T00:45:36.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034700.395556_1259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:45:00.395556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:36.698 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:36.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:36.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:45:36.699 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:45:36.699 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:45:36.699 DEBUG:boto:Method: HEAD 2023-09-30T00:45:36.699 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:45:36.699 DEBUG:boto:Data: 2023-09-30T00:45:36.700 DEBUG:boto:Headers: {} 2023-09-30T00:45:36.700 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:45:36.700 DEBUG:boto:Port: 8000 2023-09-30T00:45:36.700 DEBUG:boto:Params: {} 2023-09-30T00:45:36.700 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:45:36.701 DEBUG:boto:Token: None 2023-09-30T00:45:36.701 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:45:36 GMT /dmuhsu-89/ 2023-09-30T00:45:36.701 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kVNuXLHsNEmGGnOrM9P3V6XUPj8= 2023-09-30T00:45:36.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:45:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kVNuXLHsNEmGGnOrM9P3V6XUPj8=', 'Content-Length': '0'} 2023-09-30T00:45:36.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040042d842480fc6a-0065176fb0-28e0-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:45:36.707 DEBUG:boto:path=/dmuhsu-89/obj3 2023-09-30T00:45:36.707 DEBUG:boto:auth_path=/dmuhsu-89/obj3 2023-09-30T00:45:36.707 DEBUG:boto:Method: PUT 2023-09-30T00:45:36.708 DEBUG:boto:Path: /dmuhsu-89/obj3 2023-09-30T00:45:36.708 DEBUG:boto:Data: 2023-09-30T00:45:36.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:45:36.708 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:45:36.708 DEBUG:boto:Port: 8000 2023-09-30T00:45:36.708 DEBUG:boto:Params: {} 2023-09-30T00:45:36.708 DEBUG:boto:Token: None 2023-09-30T00:45:36.708 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:45:36 GMT /dmuhsu-89/obj3 2023-09-30T00:45:36.709 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:47Lh/hB2s+My3sbW/6rMouZFEHk= 2023-09-30T00:45:36.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:45:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:47Lh/hB2s+My3sbW/6rMouZFEHk='} 2023-09-30T00:45:36.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000502908dc2b67830-0065176fb0-28e0-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:45:36.720 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:45:36.720 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:45:36.720 DEBUG:boto:Method: HEAD 2023-09-30T00:45:36.720 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:45:36.720 DEBUG:boto:Data: 2023-09-30T00:45:36.720 DEBUG:boto:Headers: {} 2023-09-30T00:45:36.721 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:45:36.721 DEBUG:boto:Port: 8000 2023-09-30T00:45:36.721 DEBUG:boto:Params: {} 2023-09-30T00:45:36.721 DEBUG:boto:Token: None 2023-09-30T00:45:36.721 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:45:36 GMT /dmuhsu-89/ 2023-09-30T00:45:36.721 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:kVNuXLHsNEmGGnOrM9P3V6XUPj8= 2023-09-30T00:45:36.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:45:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kVNuXLHsNEmGGnOrM9P3V6XUPj8=', 'Content-Length': '0'} 2023-09-30T00:45:36.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbb920047ee85c8b-0065176fb0-28e0-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:45:36.724 DEBUG:boto:path=/dmuhsu-89/obj4 2023-09-30T00:45:36.724 DEBUG:boto:auth_path=/dmuhsu-89/obj4 2023-09-30T00:45:36.724 DEBUG:boto:Method: PUT 2023-09-30T00:45:36.724 DEBUG:boto:Path: /dmuhsu-89/obj4 2023-09-30T00:45:36.724 DEBUG:boto:Data: 2023-09-30T00:45:36.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:45:36.724 DEBUG:boto:Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:45:36.724 DEBUG:boto:Port: 8000 2023-09-30T00:45:36.725 DEBUG:boto:Params: {} 2023-09-30T00:45:36.725 DEBUG:boto:Token: None 2023-09-30T00:45:36.725 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 30 Sep 2023 00:45:36 GMT /dmuhsu-89/obj4 2023-09-30T00:45:36.725 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Nt8Qi6GfFO/kFL0FzIhunes0dMU= 2023-09-30T00:45:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:45:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Nt8Qi6GfFO/kFL0FzIhunes0dMU='} 2023-09-30T00:45:36.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036e201b50434d8e9-0065176fb0-28e0-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:45:36.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.771 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:36.817 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:36.817 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:36.858 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.879 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:45:00.395556Z" 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:36.880 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:36.881 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:36.881 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:45:36.881 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:36.881 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:45:36.882 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:36.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2023-09-30T00:45:36.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.883 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:36.936 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:36.936 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:36.970 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:36.969+0000 7f81c1f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:36.970 INFO:teuthology.orchestra.run.smithi184.stderr:2023-09-30T00:45:36.970+0000 7f81c1f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:36.985 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:36.985 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:36.986 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:36.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:36.988 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:36.989 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:45:00.395556Z", 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:36.990 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2023-09-30T00:45:36.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:36Z" 2023-09-30T00:45:36.992 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:36.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034700.395556_1259.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:45:00.395556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:36Z" } 2023-09-30T00:45:36.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034700.395556_1259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:45:00.395556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:36.994 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:36.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:36.994 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2023-09-30T00:45:36.994 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2023-09-30T00:45:36.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:36.995 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:37.041 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:37.041 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:37.090+0000 7f02d534b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:37.091 INFO:teuthology.orchestra.run.smithi175.stderr:2023-09-30T00:45:37.091+0000 7f02d534b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:45:37.105 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:37.106 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:37.107 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.108 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696034700.395556_1259.1", 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-30T00:45:00.395556Z", 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2023-09-30T00:45:37.110 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2023-09-30T00:45:37.111 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2023-09-30T00:45:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.112 INFO:teuthology.orchestra.run.smithi175.stdout: "current_time": "2023-09-30T00:45:37Z" 2023-09-30T00:45:37.112 INFO:teuthology.orchestra.run.smithi175.stdout:} 2023-09-30T00:45:37.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696031700.274277_207.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T23:55:00.274277Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696031700.294167_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:55:00.294167Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1696034700.395556_1259.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:45:00.395556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1696025517.047167_98.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:11:57.047167Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-30T00:45:37Z" } 2023-09-30T00:45:37.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034700.395556_1259.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:45:00.395556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:45:37.113 DEBUG:rgw_multi.tests:log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:37.113 DEBUG:rgw_multi.tests:sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034700.395556_1259.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:45:37.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2023-09-30T00:45:37.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:37.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:37.114 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:37.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:37.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000013790", 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:45:36.733189Z" 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000025208", 2023-09-30T00:45:37.214 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:45:36.719833Z" 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: { 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout: } 2023-09-30T00:45:37.215 INFO:teuthology.orchestra.run.smithi175.stdout:] 2023-09-30T00:45:37.216 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:37.217 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:45:37.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:37.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:37.217 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:37.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:37.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:37.320 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:37.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:37Z" 2023-09-30T00:45:37.326 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:37.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:45:37Z" } 2023-09-30T00:45:37.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:45:37.327 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:37.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:37.328 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:45:42.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:42.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:42.329 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:42.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:42.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:42.434 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:45:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:45:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:45:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:42Z" 2023-09-30T00:45:42.441 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:42.442 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:45:42Z" } 2023-09-30T00:45:42.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:45:42.442 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:42.442 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:42.442 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:45:47.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:47.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:47.444 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:47.499 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:47.499 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:47.550 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:47.551 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:47.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:47.553 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:47.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:47.556 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:47Z" 2023-09-30T00:45:47.557 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:47.558 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:45:47Z" } 2023-09-30T00:45:47.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:45:47.559 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:47.559 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:47.559 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:45:52.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:52.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:52.560 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:52.612 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:52.612 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:52.661 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:52.661 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:52.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:45:52.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:45:52.664 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:52.665 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:52.666 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:52.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:52Z" 2023-09-30T00:45:52.668 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:52.669 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:45:52Z" } 2023-09-30T00:45:52.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:45:52.670 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:52.670 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:52.670 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:45:57.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:57.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:45:57.671 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:45:57.727 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:45:57.727 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:45:57.777 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:45:57.777 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:45:57.777 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:45:57.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:57.778 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:57.779 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:57.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:45:57.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:45:57.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:45:57Z" 2023-09-30T00:45:57.783 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:45:57.784 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:45:57Z" } 2023-09-30T00:45:57.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:45:57.784 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:57.784 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:45:57.785 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:02.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:02.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:02.786 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:02.841 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:02.842 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:02.892 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:02.892 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:02.893 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.894 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:02.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:02.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:02.897 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:02Z" 2023-09-30T00:46:02.898 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:02.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:02Z" } 2023-09-30T00:46:02.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:02.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:02.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:02.900 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:07.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:07.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:07.902 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:07.956 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:07.957 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:08.006 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:08.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:08.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:08.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:08.013 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:08.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:08.013 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:07Z" 2023-09-30T00:46:08.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:08.014 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:07Z" } 2023-09-30T00:46:08.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:08.014 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:08.014 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:08.015 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:13.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:13.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:13.016 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:13.070 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:13.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:13.122 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:13.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:13.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:13.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:13.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:13.128 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:13Z" 2023-09-30T00:46:13.129 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:13.130 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:13Z" } 2023-09-30T00:46:13.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:13.131 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:13.131 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:13.131 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:18.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:18.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:18.132 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:18.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:18.190 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:18.246 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:18.246 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:18.246 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:18.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:18.248 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:18.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:18.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:18.251 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:18.252 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:18.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:18Z" 2023-09-30T00:46:18.254 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:18.254 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:18Z" } 2023-09-30T00:46:18.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:18.255 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:18.255 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:18.255 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:23.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:23.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:23.256 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:23.312 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:23.312 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:23.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:23.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:23.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:23.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:23.367 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:23.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:23.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:23.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:23.371 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:23Z" 2023-09-30T00:46:23.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:23.372 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:23Z" } 2023-09-30T00:46:23.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:23.373 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:23.373 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:23.373 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:28.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:28.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:28.375 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:28.430 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:28.430 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:28.480 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:28.481 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:28Z" 2023-09-30T00:46:28.486 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:28.487 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:28Z" } 2023-09-30T00:46:28.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:28.488 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:28.488 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:28.488 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:33.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:33.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:33.489 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:33.544 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:33.544 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:33.594 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:33.594 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:33.594 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:33Z" 2023-09-30T00:46:33.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:33.601 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:33Z" } 2023-09-30T00:46:33.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:33.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:33.602 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:33.602 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:38.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:38.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:38.603 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:38.658 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:38.659 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:38.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:38.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:38Z" 2023-09-30T00:46:38.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:38.716 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:38Z" } 2023-09-30T00:46:38.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:38.717 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:38.717 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:38.717 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:43.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:43.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:43.718 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:43.773 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:43.773 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:43.824 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:43Z" 2023-09-30T00:46:43.830 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:43.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:43Z" } 2023-09-30T00:46:43.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:43.831 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:43.832 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:43.832 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:48.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:48.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:48.833 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:48.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:48.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:48.938 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:48.938 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:48.939 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:48.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:48.941 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:48.942 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:48.943 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:48.944 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:48Z" 2023-09-30T00:46:48.944 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:48.945 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:48Z" } 2023-09-30T00:46:48.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:48.945 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:48.945 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:48.945 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:53.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:53.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:53.946 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:54.053 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:54.053 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:54.053 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:54.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:54.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:54.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:54.057 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:54.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:54.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:54Z" 2023-09-30T00:46:54.060 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:54.061 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:54Z" } 2023-09-30T00:46:54.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:54.061 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:54.061 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:54.061 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:46:59.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:59.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:46:59.062 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:46:59.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:46:59.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:46:59.167 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:46:59.167 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:46:59.167 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:46:59.167 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:46:59.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:59.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025067", 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.945119Z" 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:59.170 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:46:59.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:46:59.172 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:46:59Z" 2023-09-30T00:46:59.173 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:46:59.174 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013650", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:43:02.909341Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025067", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:43:25.945119Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:46:59Z" } 2023-09-30T00:46:59.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025067', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.945119Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:46:59.174 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:59.175 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000025067', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:46:59.175 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013790 target=00000000000000000000:00000000000000013650, shard=1 master=00000000000000000000:00000000000000025208 target=00000000000000000000:00000000000000025067 2023-09-30T00:47:04.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.176 DEBUG:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2023-09-30T00:47:04.281 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2023-09-30T00:47:04.281 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2023-09-30T00:47:04.281 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2023-09-30T00:47:04.281 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5247795594334234800 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:47:04.282 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013790", 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 9, 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:45:36.733189Z" 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000025208", 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 66, 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:45:36.719833Z" 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:47:04.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:04.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018894", 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2023-09-30T00:47:04.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2023-09-30T00:43:25.974931Z" 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 75, 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 75 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout: "current_time": "2023-09-30T00:47:04Z" 2023-09-30T00:47:04.287 INFO:teuthology.orchestra.run.smithi184.stdout:} 2023-09-30T00:47:04.288 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5247795594334234800 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013790", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-30T00:45:36.733189Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025208", "next_step_marker": "", "total_entries": 66, "pos": 0, "timestamp": "2023-09-30T00:45:36.719833Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021828", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:32:32.326318Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018894", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-30T00:43:25.974931Z" } } ] }, "full_sync": { "total": 75, "complete": 75 }, "current_time": "2023-09-30T00:47:04Z" } 2023-09-30T00:47:04.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013790', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:45:36.733189Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025208', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:45:36.719833Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018894', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:25.974931Z'}}] 2023-09-30T00:47:04.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:47:04.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013790', 1: '00000000000000000000:00000000000000025208', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018894'} 2023-09-30T00:47:04.289 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:04.289 DEBUG:boto:path=/dmuhsu-89/ 2023-09-30T00:47:04.289 DEBUG:boto:auth_path=/dmuhsu-89/ 2023-09-30T00:47:04.290 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.290 DEBUG:boto:Path: /dmuhsu-89/ 2023-09-30T00:47:04.290 DEBUG:boto:Data: 2023-09-30T00:47:04.290 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.290 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.290 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.290 DEBUG:boto:Params: {} 2023-09-30T00:47:04.290 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:04.291 DEBUG:boto:Token: None 2023-09-30T00:47:04.291 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-89/ 2023-09-30T00:47:04.291 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:Vli15HacRMj+j1z1c+9T6MdUk1c= 2023-09-30T00:47:04.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Vli15HacRMj+j1z1c+9T6MdUk1c=', 'Content-Length': '0'} 2023-09-30T00:47:04.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000348003862ecf9187-0065177008-220a-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.296 DEBUG:boto:path=/dmuhsu-89/obj3 2023-09-30T00:47:04.296 DEBUG:boto:auth_path=/dmuhsu-89/obj3 2023-09-30T00:47:04.297 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.297 DEBUG:boto:Path: /dmuhsu-89/obj3 2023-09-30T00:47:04.297 DEBUG:boto:Data: 2023-09-30T00:47:04.297 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.297 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.297 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.297 DEBUG:boto:Params: {} 2023-09-30T00:47:04.298 DEBUG:boto:Token: None 2023-09-30T00:47:04.298 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-89/obj3 2023-09-30T00:47:04.298 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:iutT4cq94MgXNDapFhkShMJ6lMs= 2023-09-30T00:47:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iutT4cq94MgXNDapFhkShMJ6lMs=', 'Content-Length': '0'} 2023-09-30T00:47:04.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008474bba785c74853-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.300 DEBUG:boto:path=/dmuhsu-89/obj3 2023-09-30T00:47:04.301 DEBUG:boto:auth_path=/dmuhsu-89/obj3 2023-09-30T00:47:04.301 DEBUG:boto:Method: GET 2023-09-30T00:47:04.301 DEBUG:boto:Path: /dmuhsu-89/obj3 2023-09-30T00:47:04.301 DEBUG:boto:Data: 2023-09-30T00:47:04.301 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.301 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.302 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.302 DEBUG:boto:Params: {} 2023-09-30T00:47:04.302 DEBUG:boto:Token: None 2023-09-30T00:47:04.302 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-89/obj3 2023-09-30T00:47:04.302 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:uM2P2fBEAXffZeM3ZFO/bOLZD1k= 2023-09-30T00:47:04.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uM2P2fBEAXffZeM3ZFO/bOLZD1k=', 'Content-Length': '0'} 2023-09-30T00:47:04.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009551f96a9c55fcfd-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.304 DEBUG:boto:path=/dmuhsu-89/obj4 2023-09-30T00:47:04.305 DEBUG:boto:auth_path=/dmuhsu-89/obj4 2023-09-30T00:47:04.305 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.305 DEBUG:boto:Path: /dmuhsu-89/obj4 2023-09-30T00:47:04.305 DEBUG:boto:Data: 2023-09-30T00:47:04.305 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.305 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.305 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.305 DEBUG:boto:Params: {} 2023-09-30T00:47:04.305 DEBUG:boto:Token: None 2023-09-30T00:47:04.306 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-89/obj4 2023-09-30T00:47:04.306 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:wLCuEsoAu+UNUdnxt8sKtEJmP2o= 2023-09-30T00:47:04.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wLCuEsoAu+UNUdnxt8sKtEJmP2o=', 'Content-Length': '0'} 2023-09-30T00:47:04.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4bc66a106fdca4c-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.308 DEBUG:boto:path=/dmuhsu-89/obj4 2023-09-30T00:47:04.308 DEBUG:boto:auth_path=/dmuhsu-89/obj4 2023-09-30T00:47:04.308 DEBUG:boto:Method: GET 2023-09-30T00:47:04.308 DEBUG:boto:Path: /dmuhsu-89/obj4 2023-09-30T00:47:04.308 DEBUG:boto:Data: 2023-09-30T00:47:04.308 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.309 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.309 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.309 DEBUG:boto:Params: {} 2023-09-30T00:47:04.309 DEBUG:boto:Token: None 2023-09-30T00:47:04.309 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-89/obj4 2023-09-30T00:47:04.309 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:OcOzJIBp8Ud7dgm03XQo9ZbsTds= 2023-09-30T00:47:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OcOzJIBp8Ud7dgm03XQo9ZbsTds=', 'Content-Length': '0'} 2023-09-30T00:47:04.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a73d0d4c957674b-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.312 DEBUG:boto:path=/dmuhsu-90/ 2023-09-30T00:47:04.312 DEBUG:boto:auth_path=/dmuhsu-90/ 2023-09-30T00:47:04.312 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.312 DEBUG:boto:Path: /dmuhsu-90/ 2023-09-30T00:47:04.312 DEBUG:boto:Data: 2023-09-30T00:47:04.312 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.312 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.312 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.313 DEBUG:boto:Params: {} 2023-09-30T00:47:04.313 DEBUG:boto:Token: None 2023-09-30T00:47:04.313 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-90/ 2023-09-30T00:47:04.313 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:vsxANnLurTCYRMntlunyPIOq82M= 2023-09-30T00:47:04.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vsxANnLurTCYRMntlunyPIOq82M=', 'Content-Length': '0'} 2023-09-30T00:47:04.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000857d6fec468a72af-0065177008-220a-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.315 DEBUG:boto:path=/dmuhsu-90/obj3 2023-09-30T00:47:04.315 DEBUG:boto:auth_path=/dmuhsu-90/obj3 2023-09-30T00:47:04.315 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.315 DEBUG:boto:Path: /dmuhsu-90/obj3 2023-09-30T00:47:04.315 DEBUG:boto:Data: 2023-09-30T00:47:04.315 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.315 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.316 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.316 DEBUG:boto:Params: {} 2023-09-30T00:47:04.316 DEBUG:boto:Token: None 2023-09-30T00:47:04.316 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-90/obj3 2023-09-30T00:47:04.316 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:r9dubHGxVM2/sQDkzJPsSWUsPC8= 2023-09-30T00:47:04.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r9dubHGxVM2/sQDkzJPsSWUsPC8=', 'Content-Length': '0'} 2023-09-30T00:47:04.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef40dbe6d94a8d52-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.318 DEBUG:boto:path=/dmuhsu-90/obj3 2023-09-30T00:47:04.318 DEBUG:boto:auth_path=/dmuhsu-90/obj3 2023-09-30T00:47:04.318 DEBUG:boto:Method: GET 2023-09-30T00:47:04.318 DEBUG:boto:Path: /dmuhsu-90/obj3 2023-09-30T00:47:04.318 DEBUG:boto:Data: 2023-09-30T00:47:04.318 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.318 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.318 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.319 DEBUG:boto:Params: {} 2023-09-30T00:47:04.319 DEBUG:boto:Token: None 2023-09-30T00:47:04.319 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-90/obj3 2023-09-30T00:47:04.319 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hCTZ93QMQ9w51vxKNb8RXIr4Qkw= 2023-09-30T00:47:04.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hCTZ93QMQ9w51vxKNb8RXIr4Qkw=', 'Content-Length': '0'} 2023-09-30T00:47:04.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d43b5aee38b3bea0-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.362 DEBUG:boto:path=/dmuhsu-90/obj4 2023-09-30T00:47:04.363 DEBUG:boto:auth_path=/dmuhsu-90/obj4 2023-09-30T00:47:04.363 DEBUG:boto:Method: HEAD 2023-09-30T00:47:04.363 DEBUG:boto:Path: /dmuhsu-90/obj4 2023-09-30T00:47:04.363 DEBUG:boto:Data: 2023-09-30T00:47:04.363 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.363 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.363 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.363 DEBUG:boto:Params: {} 2023-09-30T00:47:04.364 DEBUG:boto:Token: None 2023-09-30T00:47:04.364 DEBUG:boto:StringToSign: HEAD Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-90/obj4 2023-09-30T00:47:04.364 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:7oT5nRnI5TLwX24vZ/opi5JhQjY= 2023-09-30T00:47:04.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:7oT5nRnI5TLwX24vZ/opi5JhQjY=', 'Content-Length': '0'} 2023-09-30T00:47:04.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054dd048ca13893c1-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.366 DEBUG:boto:path=/dmuhsu-90/obj4 2023-09-30T00:47:04.366 DEBUG:boto:auth_path=/dmuhsu-90/obj4 2023-09-30T00:47:04.366 DEBUG:boto:Method: GET 2023-09-30T00:47:04.366 DEBUG:boto:Path: /dmuhsu-90/obj4 2023-09-30T00:47:04.366 DEBUG:boto:Data: 2023-09-30T00:47:04.366 DEBUG:boto:Headers: {} 2023-09-30T00:47:04.367 DEBUG:boto:Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.367 DEBUG:boto:Port: 8000 2023-09-30T00:47:04.367 DEBUG:boto:Params: {} 2023-09-30T00:47:04.367 DEBUG:boto:Token: None 2023-09-30T00:47:04.367 DEBUG:boto:StringToSign: GET Sat, 30 Sep 2023 00:47:04 GMT /dmuhsu-90/obj4 2023-09-30T00:47:04.367 DEBUG:boto:Signature: AWS PTXB0TDQIX657E2O5ERJ:hD9CdmWsWcB5MpWAI0r8OzyqmBk= 2023-09-30T00:47:04.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:47:04 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hD9CdmWsWcB5MpWAI0r8OzyqmBk=', 'Content-Length': '0'} 2023-09-30T00:47:04.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:45:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-89:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.10046.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000624e6d568ca08b95-0065177008-220a-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:47:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:04.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dmuhsu-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:04.369 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dmuhsu-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:04.423 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.424 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.492 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:47:04.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:04.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:04.492 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:04.536 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.537 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.602 INFO:teuthology.orchestra.run.smithi175.stdout:{"sync_policy":{"groups":[]}} 2023-09-30T00:47:04.603 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_encrypted_object_sync 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:04.604 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1286, in test_encrypted_object_sync 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests: eq(data, key.get_contents_as_string(headers=sse_c_headers, encoding='ascii')) 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:AttributeError: 'NoneType' object has no attribute 'get_contents_as_string' 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-64 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-64/ 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-64/ 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-64/ 2023-09-30T00:47:04.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.606 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:28 GMT 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:/dmuhsu-64/ 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NdSvXSTbG7bNJq18Cn+u/M0JYrU= 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NdSvXSTbG7bNJq18Cn+u/M0JYrU=', 'Content-Length': '0'} 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c8a0cdde47d77a73-0065175bbc-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-30T00:47:04.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:28 GMT 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2023-09-30T00:47:04.608 INFO:tasks.rgw_multisite_tests:/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:xBIVKScy6GZNH8IVvB7ESPmzeY8= 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xBIVKScy6GZNH8IVvB7ESPmzeY8='} 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000dc20ca3768f84cd3-0065175bbc-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-64/testobj-sse-kms 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-64/testobj-sse-kms 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-64/testobj-sse-kms 2023-09-30T00:47:04.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.610 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:28 GMT 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:/dmuhsu-64/testobj-sse-kms 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Zw7kcb/VXu50IZfheVzU7CR94xo= 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Zw7kcb/VXu50IZfheVzU7CR94xo='} 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZG11aHN1LTY0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000072751116fd97734b-0065175bbc-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029628.863421_509.1", 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:20:28.863421Z" 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:20:29.207+0000 7f7738b11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:20:29.208+0000 7f7738b11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:04.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:04.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696029628.863421_509.1", 2023-09-30T00:47:04.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-30T00:47:04.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:04.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:04.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:04.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:04.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.626 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.626 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.626 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.627 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.629 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.629 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.629 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.630 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.630 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.630 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.631 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029628.863421_509.1", 2023-09-30T00:47:04.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-30T00:47:04.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.633 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.634 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.634 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.634 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.634 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.636 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029628.863421_509.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:28.863421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:04.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:04.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:29.332+0000 7f0fc3044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:29.333+0000 7f0fc3044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029628.863421_509.1", 2023-09-30T00:47:04.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:04.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.652 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.653 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.654 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.655 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029628.863421_509.1", 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:20:28.863421Z", 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.657 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.659 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029628.863421_509.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:28.863421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029628.863421_509.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-64 2023-09-30T00:47:04.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-64/ 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-64/ 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-64/ 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:/dmuhsu-64/ 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:TgQlr0B3/CQgKsZ0oDSUOr4jXxw= 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TgQlr0B3/CQgKsZ0oDSUOr4jXxw=', 'Content-Length': '0'} 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fe9d3f2010cce65-0065175bbd-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.663 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:/dmuhsu-64/testobj-sse-c 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+PEj6Ra1+D/WbbwyopcPKn49tdk= 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+PEj6Ra1+D/WbbwyopcPKn49tdk=', 'Content-Length': '0'} 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000159a37d1a924aedd-0065175bbd-14ab-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.664 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:04.665 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2023-09-30T00:47:04.665 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:04.666 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:04.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:04.666 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:04.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 867, in test_bucket_delete_notempty 2023-09-30T00:47:04.666 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-36 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-36/ 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-36/ 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-36/ 2023-09-30T00:47:04.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:28:05 GMT 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:/dmuhsu-36/ 2023-09-30T00:47:04.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:SreNg293zZtVH9eeArZXWkXP8tg= 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SreNg293zZtVH9eeArZXWkXP8tg=', 'Content-Length': '0'} 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008a2aefa3e90c8aac-0065174f75-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-37 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-37/ 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-37/ 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-37/ 2023-09-30T00:47:04.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:04.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:04.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:04.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:28:05 GMT 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:/dmuhsu-37/ 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:U6hTIqSJfX9k9sipQPJDHBxwqGI= 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:U6hTIqSJfX9k9sipQPJDHBxwqGI=', 'Content-Length': '0'} 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002fa8ce65286a0c60-0065174f75-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-38 2023-09-30T00:47:04.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-38/ 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-38/ 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-38/ 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:04.672 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:28:05 GMT 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:/dmuhsu-38/ 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bPr54LNQJVl4OQY2mW5OFJ9aRoM= 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:05 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bPr54LNQJVl4OQY2mW5OFJ9aRoM=', 'Content-Length': '0'} 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003a025d187a533161-0065174f75-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026485.350874_251.1", 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:28:05.350874Z" 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:28:05.623+0000 7fb0b589d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:28:05.624+0000 7fb0b589d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:28:05Z" 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.681 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.682 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.683 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.686 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:05Z" 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:28:10.732+0000 7f51baf2d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:28:10.733+0000 7f51baf2d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:04.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696026485.350874_251.1", 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:28:05.350874Z", 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:28:10Z" 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.693 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.694 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.695 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026485.350874_251.1", 2023-09-30T00:47:04.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:05.350874Z", 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.698 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:10Z" 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026485.350874_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:05.350874Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:10.827+0000 7f073f14b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:10.828+0000 7f073f14b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:10Z" 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.705 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.707 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.710 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:10Z" 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:15.925+0000 7fbb8fcc0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:15.926+0000 7fbb8fcc0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:15Z" 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.722 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:15Z" 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.962427_205.1 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:21.025+0000 7f15c6730880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:21.026+0000 7f15c6730880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:21Z" 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.729 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:04.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.733 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:21Z" 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:26.154+0000 7f2f54332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:26.155+0000 7f2f54332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:26Z" 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.741 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.742 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.743 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.744 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.745 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:26Z" 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:31.247+0000 7f07af026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:31.248+0000 7f07af026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:31Z" 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.755 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.756 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.757 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:31Z" 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:36.359+0000 7f2425f37880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:36.360+0000 7f2425f37880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:36Z" 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.765 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.766 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.767 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.768 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.769 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:36Z" 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:41.495+0000 7f020d727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:41.496+0000 7f020d727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:41Z" 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.779 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.780 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.781 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.782 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.783 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:41Z" 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.825442_197.1 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:46.590+0000 7fee8c88b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:46.591+0000 7fee8c88b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:46Z" 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.792 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.793 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.794 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.795 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:46Z" 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:51.687+0000 7f197251e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:51.687+0000 7f197251e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:51Z" 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.803 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.804 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.805 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.806 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.807 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:51Z" 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:56.782+0000 7fb76a92c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:28:56.783+0000 7fb76a92c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:28:56Z" 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.816 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.817 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.818 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.819 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:28:56Z" 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:01.878+0000 7f3f66c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:01.879+0000 7f3f66c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:01Z" 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.827 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.828 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.829 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.830 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.831 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:01Z" 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.905539_203.1 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:07.017+0000 7fb5f8b30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:07.018+0000 7fb5f8b30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:07Z" 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.839 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.840 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.841 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.842 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.843 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:07Z" 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:12.115+0000 7f703cd2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:12.116+0000 7f703cd2c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:12Z" 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.851 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.852 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.853 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.854 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.855 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:12Z" 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:17.213+0000 7fe56594b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:17.214+0000 7fe56594b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:17Z" 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.863 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.864 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.865 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.866 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.867 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:17Z" 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.962427_205.1 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:22.366+0000 7fd84591f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:22.367+0000 7fd84591f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:22Z" 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.875 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.876 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:04.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.879 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:22Z" 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:27.473+0000 7f7031526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:27.474+0000 7f7031526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:27Z" 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.886 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:27Z" 2023-09-30T00:47:04.891 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:32.574+0000 7f4d52175880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:32.575+0000 7f4d52175880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:32Z" 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.898 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:04.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:32Z" 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.903 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:37.669+0000 7f2c56c0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:37.670+0000 7f2c56c0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:37Z" 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.910 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.912 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.913 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.914 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:37Z" 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:42.793+0000 7f5f5b723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:42.794+0000 7f5f5b723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:42Z" 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.923 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.924 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:42Z" 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.927 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:47.937+0000 7f0d6cd27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:47.938+0000 7f0d6cd27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:47Z" 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.934 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.935 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.936 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.937 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.938 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:47Z" 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:53.035+0000 7f9d72d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:53.036+0000 7f9d72d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:04.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:53Z" 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.946 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.947 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.948 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.950 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:53Z" 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:58.135+0000 7f3ad1611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:29:58.135+0000 7f3ad1611880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:29:58Z" 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.957 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.959 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:29:58Z" 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:03.231+0000 7f667a927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:03.232+0000 7f667a927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:03Z" 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.969 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:03Z" 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.974 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:08.332+0000 7fef5d323880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:08.332+0000 7fef5d323880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:08Z" 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.982 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:04.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.986 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.987 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:08Z" 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:04.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:13.431+0000 7fa290f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:13.431+0000 7fa290f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:04.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:13Z" 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:04.994 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.995 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:04.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:13Z" 2023-09-30T00:47:04.999 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:18.526+0000 7effe471f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:18.527+0000 7effe471f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:18Z" 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.006 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.007 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.008 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.009 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.010 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:18Z" 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:23.651+0000 7f715520f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:23.652+0000 7f715520f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.963214_206.1", 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:23Z" 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.018 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.019 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.963214_206.1", 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.021 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.022 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:23Z" 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.963214_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.963214Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.963214_206.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.963214_206.1 2023-09-30T00:47:05.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:28.753+0000 7fafeab26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:28.754+0000 7fafeab26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:28Z" 2023-09-30T00:47:05.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.030 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.031 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.033 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.034 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:28Z" 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:33.861+0000 7f98ac82e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:33.862+0000 7f98ac82e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006025_213.1", 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-30T00:47:05.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:33Z" 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.041 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.043 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.044 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006025_213.1", 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.045 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.046 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:33Z" 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006025_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006025_213.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.006025_213.1 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:38.961+0000 7fe889b34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:38.962+0000 7fe889b34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:05.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:38Z" 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.053 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:05.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.058 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:38Z" 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.960282_204.1 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:44.070+0000 7f7b9f321880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:44.071+0000 7f7b9f321880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:44Z" 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.065 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.068 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.070 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:44Z" 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:49.177+0000 7f0db354b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:49.178+0000 7f0db354b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:49Z" 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.077 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.078 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.082 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:49Z" 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:54.277+0000 7f3e1d533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:54.278+0000 7f3e1d533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:54Z" 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.089 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.091 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.094 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:54Z" 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:59.378+0000 7fa8a7c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:30:59.379+0000 7fa8a7c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:30:59Z" 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.101 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.103 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.104 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.105 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:30:59Z" 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:04.533+0000 7ff7b0d2f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:04.539+0000 7ff7b0d2f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:04Z" 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.113 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.114 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.115 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.116 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.117 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:04Z" 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.118 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:09.657+0000 7f9eb074b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:09.658+0000 7f9eb074b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:09Z" 2023-09-30T00:47:05.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.126 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.129 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:09Z" 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:14.763+0000 7fce4e1dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:14.764+0000 7fce4e1dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:05.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:14Z" 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.136 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.138 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.141 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:14Z" 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.003827_212.1 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:19.861+0000 7f7d3b40a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:19.862+0000 7f7d3b40a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:19Z" 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.149 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.153 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:19Z" 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:24.969+0000 7f9b6dd33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:24.970+0000 7f9b6dd33880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:24Z" 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.160 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.161 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.162 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.163 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.164 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:24Z" 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:30.076+0000 7f38f66bc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:30.077+0000 7f38f66bc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:30Z" 2023-09-30T00:47:05.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.172 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.173 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.174 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.175 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.176 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:30Z" 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:05.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:35.173+0000 7fdb1e136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:35.174+0000 7fdb1e136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:35Z" 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.185 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.188 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:35Z" 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:40.283+0000 7f89a520f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:40.284+0000 7f89a520f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:40Z" 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.198 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.199 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.200 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:40Z" 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:45.441+0000 7f6dd8526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:45.442+0000 7f6dd8526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:45Z" 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.211 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.212 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:45Z" 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.213 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:50.538+0000 7f08e0c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:50.539+0000 7f08e0c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:50Z" 2023-09-30T00:47:05.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.220 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.221 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.223 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:50Z" 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:55.645+0000 7fd8e4736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:31:55.646+0000 7fd8e4736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:31:55Z" 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.231 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.233 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.236 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:31:55Z" 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:00.743+0000 7f5c1374b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:00.744+0000 7f5c1374b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:00Z" 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.243 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.244 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.245 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.248 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:00Z" 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026483.023985_216.1 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:05.849+0000 7f6c31321880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:05.850+0000 7f6c31321880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:05Z" 2023-09-30T00:47:05.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.256 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.258 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.260 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:05Z" 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:10.957+0000 7f190911f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:10.963+0000 7f190911f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:10Z" 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.268 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.269 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.271 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.272 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:10Z" 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:16.129+0000 7fb018224880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:16.131+0000 7fb018224880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:16Z" 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.279 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.284 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:16Z" 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:21.234+0000 7fbfc7612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:21.235+0000 7fbfc7612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:05.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:21Z" 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.293 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.295 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.296 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:21Z" 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.903428_202.1 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:26.360+0000 7fe0e8b26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:26.360+0000 7fe0e8b26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:26Z" 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.303 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.308 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:26Z" 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:31.464+0000 7f4239644880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:31.465+0000 7f4239644880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:31Z" 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.315 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.320 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:31Z" 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:36.572+0000 7f15a9c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:36.573+0000 7f15a9c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:36Z" 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.327 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.328 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.330 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.331 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.332 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:36Z" 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:41.672+0000 7f7dcca9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:41.673+0000 7f7dcca9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:41Z" 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.340 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.341 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.342 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.343 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.344 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:41Z" 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:46.783+0000 7ff32fb1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:46.784+0000 7ff32fb1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:46Z" 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.352 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.353 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.354 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.355 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.356 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:46Z" 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:51.887+0000 7f556fd36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:51.888+0000 7f556fd36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:51Z" 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.364 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.365 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.366 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.367 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:51Z" 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.980766_211.1 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:56.986+0000 7ff8df326880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:32:56.986+0000 7ff8df326880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:32:56Z" 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.377 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:32:56Z" 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:02.085+0000 7fbe6c94b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:02.086+0000 7fbe6c94b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:02Z" 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.391 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.395 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:02Z" 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:07.217+0000 7ffb5f030880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:07.218+0000 7ffb5f030880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:07Z" 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.402 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:07Z" 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:12.317+0000 7f9eafe26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:12.318+0000 7f9eafe26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:12Z" 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.414 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.415 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.419 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:12Z" 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026485.350874_251.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026485.350874_251.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:05.420 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.421 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:05.421 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2023-09-30T00:47:05.421 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_multi_period_incremental_sync 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-09-30T00:47:05.422 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-39 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-39/ 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-39/ 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-39/ 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:05.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:33:17 GMT 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:/dmuhsu-39/ 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:V81CD62pIBU2hNpYIKCRA1nTdOY= 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:V81CD62pIBU2hNpYIKCRA1nTdOY=', 'Content-Length': '0'} 2023-09-30T00:47:05.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009a4cdb61ee7cfa24-00651750ad-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-40 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-40/ 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-40/ 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-40/ 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:05.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:33:17 GMT 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:/dmuhsu-40/ 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:BFvsyt+DnI/yh4oCKZP+Fj391zs= 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BFvsyt+DnI/yh4oCKZP+Fj391zs=', 'Content-Length': '0'} 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000274f94e096309603-00651750ad-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-41 2023-09-30T00:47:05.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-41/ 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-41/ 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-41/ 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:05.427 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:33:17 GMT 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:/dmuhsu-41/ 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QJ4NmV5ryMFRkFPlrP4zuJGs1KM= 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:17 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QJ4NmV5ryMFRkFPlrP4zuJGs1KM=', 'Content-Length': '0'} 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e6729d2a4391aaa7-00651750ad-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026797.418167_275.1", 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:33:17.418167Z" 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:33:17.607+0000 7f0f32526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:33:17.608+0000 7f0f32526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696026797.418167_275.1", 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:33:17.418167Z", 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:33:17Z" 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.436 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.437 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026797.418167_275.1", 2023-09-30T00:47:05.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:33:17.418167Z", 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.440 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.441 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:17Z" 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026797.418167_275.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:17.418167Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:17.711+0000 7f28afe8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:17.712+0000 7f28afe8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:17Z" 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.448 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.453 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:17Z" 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:22.811+0000 7f4223b31880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:22.812+0000 7f4223b31880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:22Z" 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.463 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.464 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:22Z" 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:27.918+0000 7fe2fc932880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:27.919+0000 7fe2fc932880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:27Z" 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.472 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.473 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.474 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.475 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.476 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:27Z" 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:33.028+0000 7f9635d22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:33.028+0000 7f9635d22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:05.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:33Z" 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.484 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.485 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.486 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.487 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.488 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:33Z" 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:38.126+0000 7f238f612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:38.127+0000 7f238f612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:38Z" 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.496 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.497 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.498 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.499 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.500 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:38Z" 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:43.225+0000 7f30a0d1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:43.226+0000 7f30a0d1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:43Z" 2023-09-30T00:47:05.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.508 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.509 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.510 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.511 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.512 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:43Z" 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:48.348+0000 7f324e525880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:48.349+0000 7f324e525880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:48Z" 2023-09-30T00:47:05.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.520 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.521 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.522 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.523 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.524 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:48Z" 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:53.450+0000 7fc845b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:53.451+0000 7fc845b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:53Z" 2023-09-30T00:47:05.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.532 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.533 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.534 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.535 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.536 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:53Z" 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.537 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:58.559+0000 7f0dd97dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:33:58.560+0000 7f0dd97dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:33:58Z" 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.543 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.548 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:33:58Z" 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:03.805+0000 7fa629b2a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:03.806+0000 7fa629b2a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:03Z" 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.555 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:05.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.560 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:03Z" 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.842044_199.1 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:08.921+0000 7f31fb815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:08.922+0000 7f31fb815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:08Z" 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.567 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.569 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:08Z" 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:14.043+0000 7feec7c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:14.044+0000 7feec7c12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:14Z" 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.579 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.580 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.581 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.582 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.583 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:14Z" 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.842044_199.1 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:19.174+0000 7fad83727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:19.175+0000 7fad83727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:19Z" 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.593 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.594 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.595 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.596 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:19Z" 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:24.274+0000 7f9849c67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:24.275+0000 7f9849c67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:24Z" 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.604 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.605 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.606 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.607 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.608 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:24Z" 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:05.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:29.371+0000 7f5280830880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:29.372+0000 7f5280830880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:29Z" 2023-09-30T00:47:05.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.616 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.617 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.618 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.619 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.620 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:29Z" 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.825442_197.1 2023-09-30T00:47:05.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:34.488+0000 7f698b69d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:34.489+0000 7f698b69d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:34Z" 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.632 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:34Z" 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:39.672+0000 7fc99c74b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:39.674+0000 7fc99c74b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:39Z" 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.640 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:39Z" 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:44.799+0000 7f2a92126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:44.800+0000 7f2a92126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:44Z" 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:44Z" 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:49.906+0000 7f391e015880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:49.907+0000 7f391e015880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:49Z" 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.664 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.665 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.666 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.667 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:49Z" 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:55.033+0000 7fb1a2811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:34:55.034+0000 7fb1a2811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:34:55Z" 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.675 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.676 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.677 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.678 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.679 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:34:55Z" 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:00.142+0000 7f84ca812880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:00.143+0000 7f84ca812880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:00Z" 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.687 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.688 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.689 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.690 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.691 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:00Z" 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:05.248+0000 7fab3a711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:05.249+0000 7fab3a711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:05Z" 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.699 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.700 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.701 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.702 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.703 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:05Z" 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:10.363+0000 7f182934b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:10.364+0000 7f182934b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:10Z" 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.710 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.715 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:10Z" 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:15.554+0000 7f2b82a8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:15.555+0000 7f2b82a8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:15Z" 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.725 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:15Z" 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:20.664+0000 7f7a7ac22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:20.665+0000 7f7a7ac22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:20Z" 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.734 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.735 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.736 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.737 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.738 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:20Z" 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:25.776+0000 7f097cbaf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:25.777+0000 7f097cbaf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:25Z" 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.746 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.748 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:25Z" 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.751 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:30.907+0000 7f1efbb20880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:30.908+0000 7f1efbb20880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:30Z" 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.758 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.760 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:30Z" 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.763 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:36.038+0000 7ff38cf4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:36.039+0000 7ff38cf4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:36Z" 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.771 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:36Z" 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.825442_197.1 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:41.151+0000 7f1ed831e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:41.152+0000 7f1ed831e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:41Z" 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.782 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.783 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.785 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.786 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:41Z" 2023-09-30T00:47:05.787 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:46.266+0000 7f6139d24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:46.267+0000 7f6139d24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:46Z" 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.795 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:46Z" 2023-09-30T00:47:05.800 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:51.450+0000 7f6ccbd29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:51.451+0000 7f6ccbd29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:51Z" 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.807 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.809 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:51Z" 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.812 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.962427_205.1 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:56.562+0000 7f6705c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:35:56.563+0000 7f6705c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:35:56Z" 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.819 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.820 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.821 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.822 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:35:56Z" 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:01.677+0000 7f4636922880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:01.678+0000 7f4636922880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:01Z" 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.831 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.832 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.833 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.834 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.835 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:01Z" 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:06.779+0000 7fbd12b23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:06.780+0000 7fbd12b23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:06Z" 2023-09-30T00:47:05.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.847 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:06Z" 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:12.042+0000 7f4772015880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:12.043+0000 7f4772015880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:12Z" 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.855 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.857 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.858 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.859 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:12Z" 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:17.327+0000 7fa2e2810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:17.328+0000 7fa2e2810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:17Z" 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.867 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.868 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.869 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.870 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.871 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:17Z" 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:22.439+0000 7fc36e815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:22.440+0000 7fc36e815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:22Z" 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.879 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.880 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.881 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.882 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.883 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:22Z" 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:27.579+0000 7faa3afa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:27.580+0000 7faa3afa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:27Z" 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.891 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.893 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.894 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:27Z" 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.896 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:32.706+0000 7f832d325880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:32.707+0000 7f832d325880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:32Z" 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.903 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.905 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:32Z" 2023-09-30T00:47:05.908 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:37.969+0000 7fe76dd26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:37.970+0000 7fe76dd26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:37Z" 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:37Z" 2023-09-30T00:47:05.920 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:43.155+0000 7f06a2a19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:43.155+0000 7f06a2a19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:43Z" 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.927 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:43Z" 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.932 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:48.282+0000 7fdc30936880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:48.283+0000 7fdc30936880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:48Z" 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:05.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.943 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:48Z" 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:53.391+0000 7f24b652f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:53.392+0000 7f24b652f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:53Z" 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:05.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:53Z" 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:58.501+0000 7fc388fa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:36:58.502+0000 7fc388fa2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:36:58Z" 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.963 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.964 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.965 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.966 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.967 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:36:58Z" 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:03.633+0000 7fcb66529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:03.634+0000 7fcb66529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:03Z" 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.975 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.976 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.977 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.978 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.979 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:03Z" 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.960282_204.1 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:08.745+0000 7f4574a2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:08.746+0000 7f4574a2e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:08Z" 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:05.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:05.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:08Z" 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:05.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:13.851+0000 7fb6a6c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:13.852+0000 7fb6a6c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:05.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:05.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:05.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:05.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:05.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:13Z" 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:05.999 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.000 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.001 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.002 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.003 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:13Z" 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026483.003827_212.1 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:18.972+0000 7f8a4cf1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:18.973+0000 7f8a4cf1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:18Z" 2023-09-30T00:47:06.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.011 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.012 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.013 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.014 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.015 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:18Z" 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:06.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:24.147+0000 7fd46b329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:24.154+0000 7fd46b329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:24Z" 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:24Z" 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:29.298+0000 7f756addc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:29.299+0000 7f756addc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:06.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:29Z" 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.036 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.039 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:29Z" 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:34.418+0000 7f504cf26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:34.419+0000 7f504cf26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:34Z" 2023-09-30T00:47:06.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.047 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.051 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:34Z" 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:39.540+0000 7fa821126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:39.541+0000 7fa821126880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:39Z" 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.058 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.059 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.060 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.061 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.063 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:39Z" 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:44.695+0000 7f79fd529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:44.696+0000 7f79fd529880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:44Z" 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.070 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.071 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.072 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.075 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:44Z" 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:49.808+0000 7f22b21a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:49.809+0000 7f22b21a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:49Z" 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.082 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.084 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:49Z" 2023-09-30T00:47:06.087 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:54.934+0000 7f6ab9abc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:37:54.935+0000 7f6ab9abc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:37:54Z" 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.094 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.096 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.099 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:37:54Z" 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:00.093+0000 7f6f99210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:00.097+0000 7f6f99210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:00Z" 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.106 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.108 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.111 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:00Z" 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:05.242+0000 7f344642e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:05.243+0000 7f344642e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:05Z" 2023-09-30T00:47:06.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.121 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.122 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:05Z" 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:10.361+0000 7f4adef22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:10.361+0000 7f4adef22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:10Z" 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.132 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.133 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.134 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.135 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:10Z" 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:06.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:15.490+0000 7f15e740a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:15.491+0000 7f15e740a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:15Z" 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.143 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.144 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.145 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.146 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.147 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:15Z" 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:20.626+0000 7f21cafaf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:20.631+0000 7f21cafaf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:20Z" 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.155 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.156 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.157 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:20Z" 2023-09-30T00:47:06.160 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026797.418167_275.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696026797.418167_275.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.161 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:06.162 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2023-09-30T00:47:06.162 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 985, in test_datalog_autotrim 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-09-30T00:47:06.163 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-42 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-42/ 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-42/ 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-42/ 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:/dmuhsu-42/ 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.165 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mS8SwebejWh2hJKXzZd0gRzIxfM= 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mS8SwebejWh2hJKXzZd0gRzIxfM=', 'Content-Length': '0'} 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000357eb9b8cfac14a5-00651751e1-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-43 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-43/ 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-43/ 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-43/ 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:06.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:/dmuhsu-43/ 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:wTP+3Lu1wTWZLQywbEEmoX3UVEs= 2023-09-30T00:47:06.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wTP+3Lu1wTWZLQywbEEmoX3UVEs=', 'Content-Length': '0'} 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008ecf79f773654612-00651751e1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-44 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-44/ 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-44/ 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-44/ 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:06.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:/dmuhsu-44/ 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:DfVTztQPbtQq1dtr4ylj8su/YGM= 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DfVTztQPbtQq1dtr4ylj8su/YGM=', 'Content-Length': '0'} 2023-09-30T00:47:06.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a609a526ee4f69c6-00651751e1-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-42/ 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-42/ 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-42/ 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.170 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:/dmuhsu-42/ 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Uwgb6JNU3T/GT8GE/IZxx8XO4jU= 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Uwgb6JNU3T/GT8GE/IZxx8XO4jU=', 'Content-Length': '0'} 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002961ea9b915c1131-00651751e1-1104-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-42/key 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-42/key 2023-09-30T00:47:06.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-42/key 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:06.172 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:/dmuhsu-42/key 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:SpONFyfpUrHjJLBk3/TLY3AKHuA= 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SpONFyfpUrHjJLBk3/TLY3AKHuA='} 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acf6d2764d0e385c-00651751e1-1104-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-43/ 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-43/ 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-43/ 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:/dmuhsu-43/ 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.174 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:iGftnf76Bk6nFZPgqL90llq6HYg= 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iGftnf76Bk6nFZPgqL90llq6HYg=', 'Content-Length': '0'} 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33a30cd95cf0f86-00651751e1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-43/key 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-43/key 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-43/key 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:/dmuhsu-43/key 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tXjfugpsiuw1S06lZyOvRy3ZSzk= 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tXjfugpsiuw1S06lZyOvRy3ZSzk='} 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa72960c07eb567f-00651751e1-10d7-test-zone2'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-44/ 2023-09-30T00:47:06.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-44/ 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-44/ 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.177 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:/dmuhsu-44/ 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:TWMm+xuK6xGnqJRg3ryHlnGCQ/o= 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TWMm+xuK6xGnqJRg3ryHlnGCQ/o=', 'Content-Length': '0'} 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000892e8168f3ac7b9b-00651751e1-110a-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-44/key 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-44/key 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-44/key 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:06.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:38:25 GMT 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:/dmuhsu-44/key 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:06.179 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:AnTTmOhWKQjlvd60w9F9HDzMEII= 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AnTTmOhWKQjlvd60w9F9HDzMEII='} 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e1de1675efc7524-00651751e1-110a-test-zone3'), ('Date', 'Fri, 29 Sep 2023 22:38:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:06.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:38:25.772402Z" 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:38:26.092+0000 7fc39191e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:38:26.093+0000 7fc39191e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.715044_276.1", 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.715044Z", 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:06.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:38:26Z" 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.188 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.189 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.715044_276.1", 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.715044Z", 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:26Z" 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.193 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.715044_276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.715044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.715044_276.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696027105.715044_276.1 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:38:31.201+0000 7f7a3c813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:38:31.203+0000 7f7a3c813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:38:31Z" 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.200 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.201 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.205 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:31Z" 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:31.321+0000 7f52dd34b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:31.322+0000 7f52dd34b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:31Z" 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.212 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.217 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:31Z" 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:36.433+0000 7fce04527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:36.434+0000 7fce04527880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:36Z" 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.225 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.229 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:36Z" 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi175.stdout: INFO: 2023-09-29T22:38:38.705+0000 7f303f796640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi175.stdout: INFO: 2023-09-29T22:38:39.932+0000 7f303df93640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:41.629+0000 7fd09b336880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:41.630+0000 7fd09b336880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:41Z" 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.236 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.238 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.240 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.241 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:41Z" 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:46.763+0000 7f0a69467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:46.764+0000 7f0a69467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:46Z" 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.248 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.250 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:46Z" 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.253 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:51.884+0000 7f501f330880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:51.885+0000 7f501f330880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:51Z" 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.260 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.262 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.264 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:51Z" 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:57.006+0000 7fa116e8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:38:57.007+0000 7fa116e8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:38:57Z" 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.272 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.273 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.274 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.275 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.276 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:38:57Z" 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:02.125+0000 7f2437727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:02.125+0000 7f2437727880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:02Z" 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.284 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.285 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.286 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.287 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.288 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:02Z" 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:07.241+0000 7fedacb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:07.242+0000 7fedacb26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:07Z" 2023-09-30T00:47:06.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.300 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:07Z" 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:06.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:12.365+0000 7f7bafa13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:12.366+0000 7f7bafa13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:12Z" 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.308 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.309 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.311 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.312 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:12Z" 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:17.513+0000 7ff3a6b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:17.514+0000 7ff3a6b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:06.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:17Z" 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.319 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:06.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.324 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:17Z" 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:22.637+0000 7f8995226880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:22.638+0000 7f8995226880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:22Z" 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.331 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.333 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:22Z" 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.336 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:27.778+0000 7fedd994b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:27.782+0000 7fedd994b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:27Z" 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.345 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.346 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.347 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:27Z" 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:32.930+0000 7f4dddf27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:32.931+0000 7f4dddf27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:32Z" 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.355 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.356 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.357 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.358 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.359 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:32Z" 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:38.064+0000 7f1422815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:38.065+0000 7f1422815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:38Z" 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.366 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:38Z" 2023-09-30T00:47:06.371 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:43.190+0000 7f67263af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:43.191+0000 7f67263af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.963214_206.1", 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:43Z" 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.379 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.380 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.963214_206.1", 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.963214Z", 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.382 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:43Z" 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.963214_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.963214Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.963214_206.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.963214_206.1 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:48.383+0000 7f9626a12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:48.385+0000 7f9626a12880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:48Z" 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.390 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.393 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.394 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:48Z" 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:53.525+0000 7fc9de810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:53.526+0000 7fc9de810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:53Z" 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.401 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.402 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.403 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.406 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:53Z" 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:58.636+0000 7f828f210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:39:58.636+0000 7f828f210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:39:58Z" 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.417 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.418 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.420 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:39:58Z" 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:03.799+0000 7f30d0644880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:03.800+0000 7f30d0644880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:03Z" 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.428 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.430 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.432 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:03Z" 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:08.993+0000 7fb7b6734880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:08.994+0000 7fb7b6734880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:08Z" 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.439 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.440 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.441 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.444 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:08Z" 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:14.131+0000 7f6d42226880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:14.132+0000 7f6d42226880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:14Z" 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.451 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.453 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.799906_193.1", 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.799906Z", 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.456 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:14Z" 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.799906_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.799906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.799906_193.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.799906_193.1 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:19.252+0000 7fd53df36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:19.253+0000 7fd53df36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:19Z" 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.463 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.464 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:06.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.467 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.468 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:19Z" 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.903428_202.1 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:24.371+0000 7fcddb419880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:24.372+0000 7fcddb419880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:24Z" 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.475 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.476 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.823352_196.1", 2023-09-30T00:47:06.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.823352Z", 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.479 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.480 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:24Z" 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.823352_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.823352Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.823352_196.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.823352_196.1 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:29.548+0000 7f03b5e17880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:29.549+0000 7f03b5e17880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:29Z" 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.487 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.488 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.489 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.492 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:29Z" 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:34.671+0000 7ff796120880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:34.672+0000 7ff796120880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:34Z" 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.499 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.504 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:34Z" 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:39.785+0000 7fc594a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:39.786+0000 7fc594a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:39Z" 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.511 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.513 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.516 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:39Z" 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:44.910+0000 7fcad931c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:44.910+0000 7fcad931c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:44Z" 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.528 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:44Z" 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:50.264+0000 7fdc54f22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:50.264+0000 7fdc54f22880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:06.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:50Z" 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.538 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.540 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:50Z" 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.977901_209.1 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:55.382+0000 7fe7ba267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:40:55.383+0000 7fe7ba267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:40:55Z" 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.548 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.549 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.550 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.551 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.552 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:40:55Z" 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:00.505+0000 7fd01542e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:00.506+0000 7fd01542e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:00Z" 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.560 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.561 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.562 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.563 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.564 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:00Z" 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:05.649+0000 7fb678d1a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:05.651+0000 7fb678d1a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:05Z" 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.572 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.573 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.574 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.575 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.576 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:05Z" 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:10.833+0000 7f0752067880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:10.834+0000 7f0752067880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:10Z" 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.584 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:10Z" 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.589 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:15.955+0000 7fd330336880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:15.956+0000 7fd330336880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:15Z" 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.596 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.597 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.601 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:15Z" 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:21.064+0000 7f9ef5b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:21.065+0000 7f9ef5b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:21Z" 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.608 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.609 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.610 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.796890_191.1", 2023-09-30T00:47:06.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.796890Z", 2023-09-30T00:47:06.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:21Z" 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.615 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.796890_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.796890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.796890_191.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.796890_191.1 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:26.172+0000 7f8d44a0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:26.172+0000 7f8d44a0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:26Z" 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.622 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.623 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.624 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.625 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.626 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:26Z" 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:32.531+0000 7f95af1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:32.532+0000 7f95af1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:32Z" 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:06.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:32Z" 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:37.655+0000 7f7dcae0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:37.655+0000 7f7dcae0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:37Z" 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.646 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.647 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.648 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.962427_205.1", 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.962427Z", 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.649 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.650 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:37Z" 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.962427_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.962427Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.962427_205.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.962427_205.1 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:42.869+0000 7f89ec5a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:42.874+0000 7f89ec5a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:42Z" 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.658 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.659 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.660 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.661 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.662 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:42Z" 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:48.009+0000 7f2fee524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:48.010+0000 7f2fee524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:48Z" 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.670 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.671 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.672 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.674 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:48Z" 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:53.239+0000 7f96e4932880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:53.240+0000 7f96e4932880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:53Z" 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.682 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.683 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.684 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.794634_190.1", 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.794634Z", 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.685 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.686 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:53Z" 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.794634_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.794634Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.794634_190.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.794634_190.1 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:58.381+0000 7f8bd6c67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:41:58.392+0000 7f8bd6c67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:41:58Z" 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.694 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.695 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.697 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.698 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:41:58Z" 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:03.579+0000 7f562f230880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:03.580+0000 7f562f230880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:03Z" 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.708 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:03Z" 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.711 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:08.697+0000 7fe4af72c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:08.698+0000 7fe4af72c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:08Z" 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.719 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.720 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.783295_189.1", 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.783295Z", 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.722 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.723 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:08Z" 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.783295_189.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.783295Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.783295_189.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.783295_189.1 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:14.085+0000 7fbaf8e30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:14.086+0000 7fbaf8e30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:06.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:14Z" 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.731 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.822519_195.1", 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.822519Z", 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.735 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:14Z" 2023-09-30T00:47:06.736 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.822519_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.822519Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.822519_195.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.822519_195.1 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:19.269+0000 7fa7b8c19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:19.270+0000 7fa7b8c19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:19Z" 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.743 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:19Z" 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:24.399+0000 7f3ca7c13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:24.400+0000 7f3ca7c13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:24Z" 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.755 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.757 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.825442_197.1", 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.825442Z", 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:24Z" 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.825442_197.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.825442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.825442_197.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.825442_197.1 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:29.524+0000 7f76bcd36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:29.524+0000 7f76bcd36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:29Z" 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.767 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.769 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.960282_204.1", 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.960282Z", 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.771 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.772 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:29Z" 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.960282_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.960282Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.960282_204.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.960282_204.1 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:34.720+0000 7efc19626880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:34.721+0000 7efc19626880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:34Z" 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.780 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.781 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.782 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.783 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:34Z" 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:39.836+0000 7f63e9c26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:39.837+0000 7f63e9c26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:06.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:39Z" 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.792 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.793 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.794 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.795 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.796 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:39Z" 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:44.944+0000 7f1904c24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:44.945+0000 7f1904c24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:44Z" 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.804 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.805 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.806 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.807 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.808 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:44Z" 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:50.049+0000 7ff08cb36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:50.050+0000 7ff08cb36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:50Z" 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.817 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.819 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.797765_192.1", 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.797765Z", 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:50Z" 2023-09-30T00:47:06.822 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.797765_192.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.797765Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.797765_192.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.797765_192.1 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:55.326+0000 7f1806326880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:42:55.327+0000 7f1806326880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:42:55Z" 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.831 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.903428_202.1", 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.903428Z", 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.834 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:42:55Z" 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.903428_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.903428Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.903428_202.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.903428_202.1 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:00.489+0000 7f3a65699880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:00.490+0000 7f3a65699880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:00Z" 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.841 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.843 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.846 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:00Z" 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:05.629+0000 7f5dbf136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:05.629+0000 7f5dbf136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:05Z" 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.853 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.855 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.851988_201.1", 2023-09-30T00:47:06.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.851988Z", 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.857 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.858 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:05Z" 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.851988_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.851988Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.851988_201.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.851988_201.1 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:10.882+0000 7f90b6733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:10.883+0000 7f90b6733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:10Z" 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.865 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:06.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.870 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:10Z" 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:16.009+0000 7f84e5e44880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:16.010+0000 7f84e5e44880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:16Z" 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:06.877 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.878 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.879 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.880 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.905539_203.1", 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.905539Z", 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:06.881 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:06.882 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:16Z" 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.905539_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.905539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.905539_203.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.905539_203.1 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:06.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:21.129+0000 7f1c257a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:21.130+0000 7f1c257a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:06.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:06.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:06.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:07.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:21Z" 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.435 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.436 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.437 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.842044_199.1", 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.842044Z", 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.438 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.440 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:21Z" 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.842044_199.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.842044Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.842044_199.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.842044_199.1 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:26.238+0000 7f4f2c213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:26.239+0000 7f4f2c213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:07.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:07.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:26Z" 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.448 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.450 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.450 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.450 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.450 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.450 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.451 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.820374_194.1", 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.820374Z", 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.452 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:26Z" 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.820374_194.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.820374Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.820374_194.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.820374_194.1 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:31.453+0000 7fc905723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:31.454+0000 7fc905723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:07.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:07.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:07.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:07.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:07.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:31Z" 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.868 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.869 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.870 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.839800_198.1", 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.839800Z", 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.871 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.872 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:31Z" 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.839800_198.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.839800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.839800_198.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.839800_198.1 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:36.570+0000 7f439e212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:43:36.571+0000 7f439e212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:07.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:43:36Z" 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.880 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.881 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.882 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.883 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.849856_200.1", 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.849856Z", 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.885 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:43:36Z" 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.849856_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.849856Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.849856_200.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027105.772402_287.1 target=1_1696026482.849856_200.1 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:07.887 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2023-09-30T00:47:07.888 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1028, in test_multi_zone_redirect 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-09-30T00:47:07.889 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone3 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:07.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "test-zone2", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "domain_root": "test-zone2.rgw.meta:root", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "control_pool": "test-zone2.rgw.control", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "gc_pool": "test-zone2.rgw.log:gc", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "lc_pool": "test-zone2.rgw.log:lc", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_pool": "test-zone2.rgw.log", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "intent_log_pool": "test-zone2.rgw.log:intent", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "usage_log_pool": "test-zone2.rgw.log:usage", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "roles_pool": "test-zone2.rgw.meta:roles", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "reshard_pool": "test-zone2.rgw.log:reshard", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_email_pool": "test-zone2.rgw.meta:users.email", 2023-09-30T00:47:07.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "otp_pool": "test-zone2.rgw.otp", 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "system_key": { 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "access_key": "HXNE2CFXHY4T6VO3", 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "secret_key": "0At0HHaplw6uG1twoVspt6vkyBiGHNxa" 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "placement_pools": [ 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": "default-placement", 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "index_pool": "test-zone2.rgw.buckets.index", 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "storage_classes": { 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "STANDARD": { 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_pool": "test-zone2.rgw.buckets.data", 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "compression_type": "random" 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "index_type": 0, 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "inline_data": true 2023-09-30T00:47:07.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "notif_pool": "test-zone2.rgw.log:notif" 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Sending period to new master zone d5be89e5-4ad9-4a27-8129-17eff73a6a3e 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "epoch": 4, 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": [], 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period_map": { 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "zonegroups": [ 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "test-zonegroup", 2023-09-30T00:47:07.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "test-zonegroup", 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "api_name": "", 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "is_master": true, 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "endpoints": [ 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "hostnames": [], 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "hostnames_s3website": [], 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "zones": [ 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:07.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "test-zone3", 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "endpoints": [ 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_meta": false, 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_data": true, 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "read_only": false, 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "tier_type": "", 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from": [], 2023-09-30T00:47:07.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "supported_features": [ 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "resharding" 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "test-zone1", 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "endpoints": [ 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_meta": false, 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_data": true, 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "read_only": false, 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "tier_type": "", 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from": [], 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "supported_features": [ 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "resharding" 2023-09-30T00:47:07.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "test-zone2", 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "endpoints": [ 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_meta": false, 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "log_data": true, 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "read_only": false, 2023-09-30T00:47:07.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "tier_type": "", 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from_all": false, 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_from": [], 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "supported_features": [ 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "resharding" 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "placement_targets": [ 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "name": "default-placement", 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "tags": [], 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "storage_classes": [ 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "STANDARD" 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "default_placement": "default-placement", 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_policy": { 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "groups": [] 2023-09-30T00:47:07.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled_features": [ 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "resharding" 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ], 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "short_zone_ids": [ 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": 2885950644 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": 2464975307 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": 1018467004 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "master_zonegroup": "test-zonegroup", 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period_config": { 2023-09-30T00:47:07.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket_quota": { 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled": false, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_size": -1, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_objects": -1 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_quota": { 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled": false, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_size": -1, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:07.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_objects": -1 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "user_ratelimit": { 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled": false 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket_ratelimit": { 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:07.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled": false 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "anonymous_ratelimit": { 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "enabled": false 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_name": "test-realm", 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone test-zone2 to false 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-45 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-45/ 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-45/ 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-45/ 2023-09-30T00:47:07.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:44:41 GMT 2023-09-30T00:47:07.909 INFO:tasks.rgw_multisite_tests:/dmuhsu-45/ 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:j1Jyd7whke0sLmUcCw97gnGNF0Y= 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:44:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:j1Jyd7whke0sLmUcCw97gnGNF0Y=', 'Content-Length': '0'} 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009e67ea77f7079cb6-0065175359-1abd-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:44:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-45/testredirect 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-45/testredirect 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-45/testredirect 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-30T00:47:07.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:44:41 GMT 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:/dmuhsu-45/testredirect 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qdCQMFFpqiPzQ7Dr0Mi2JGzeX4g= 2023-09-30T00:47:07.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:44:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qdCQMFFpqiPzQ7Dr0Mi2JGzeX4g='} 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000645302d0fa717290-0065175359-1abd-test-zone1'), ('Date', 'Fri, 29 Sep 2023 22:44:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:44:41.827984Z" 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:07.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:42.114+0000 7f1592b1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:42.115+0000 7f1592b1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:44:42Z" 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.920 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.923 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.925 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:44:42Z" 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:47.223+0000 7f5defe15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:47.224+0000 7f5defe15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:44:47Z" 2023-09-30T00:47:07.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.933 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.934 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.935 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.936 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.937 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:44:47Z" 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:52.331+0000 7f8b6aa10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:52.331+0000 7f8b6aa10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:44:52Z" 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.945 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:44:52Z" 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:57.439+0000 7fcd9c929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:44:57.439+0000 7fcd9c929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:44:57Z" 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.957 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:44:57Z" 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:02.551+0000 7f1fa4127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:02.552+0000 7f1fa4127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:02Z" 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.970 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.972 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.976 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:02Z" 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:07.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:07.659+0000 7fd0e8536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:07.660+0000 7fd0e8536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:07.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:07.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:07Z" 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:07.984 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.985 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.986 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.987 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:07.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:07Z" 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:07.989 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:07.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:12.767+0000 7fa11ca26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:12.768+0000 7fa11ca26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:12Z" 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.022 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:12Z" 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:17.876+0000 7fb15fc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:17.877+0000 7fb15fc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:17Z" 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.035 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.036 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.037 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.038 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:17Z" 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:22.987+0000 7fa9a2026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:22.987+0000 7fa9a2026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:22Z" 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.047 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.049 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.054 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:22Z" 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:28.096+0000 7f08275a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:28.097+0000 7f08275a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:28Z" 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.062 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.067 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:28Z" 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:33.212+0000 7f6ddc71e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:33.213+0000 7f6ddc71e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:33Z" 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.076 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.078 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.079 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.080 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.081 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:33Z" 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:38.323+0000 7fd343b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:38.324+0000 7fd343b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:38Z" 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.089 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.090 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027105.772402_287.1", 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:25.772402Z", 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:38Z" 2023-09-30T00:47:08.094 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027105.772402_287.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:25.772402Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027105.772402_287.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696027105.772402_287.1 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:43.433+0000 7f9505f34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:45:43.434+0000 7f9505f34880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:45:43Z" 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.101 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.103 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.104 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.105 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.106 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:43Z" 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:43.533+0000 7f07f9413880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:43.534+0000 7f07f9413880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:45:43Z" 2023-09-30T00:47:08.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.115 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.116 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.119 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:43Z" 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:48.656+0000 7f1c35536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:48.657+0000 7f1c35536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:45:48Z" 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.127 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.128 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.132 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:48Z" 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:53.803+0000 7f9a50444880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:53.804+0000 7f9a50444880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:45:53Z" 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.141 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.142 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.143 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.146 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:53Z" 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:58.941+0000 7fb37f136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:45:58.942+0000 7fb37f136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:45:58Z" 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.155 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.157 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.160 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:45:58Z" 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:04.105+0000 7fe8cb212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:04.106+0000 7fe8cb212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:04Z" 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.168 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.170 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.173 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:04Z" 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.174 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:09.229+0000 7f30e7e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:09.230+0000 7f30e7e15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:09Z" 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.181 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.182 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.183 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.184 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.185 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:09Z" 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:14.370+0000 7f900f0c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:14.371+0000 7f900f0c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.111037_224.1", 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:14Z" 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.193 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.195 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.111037_224.1", 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.197 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.198 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:14Z" 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.111037_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.111037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.111037_224.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.111037_224.1 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:19.550+0000 7fe41b615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:19.551+0000 7fe41b615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:19Z" 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.206 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.207 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.208 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.209 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.210 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:19Z" 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:24.670+0000 7f0470d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:24.671+0000 7f0470d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.221815_233.1", 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.221815Z", 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:24Z" 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.218 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.220 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.221815_233.1", 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.221815Z", 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.223 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:24Z" 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.221815_233.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.221815Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.221815_233.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.221815_233.1 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:29.787+0000 7fa0b9311880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:29.788+0000 7fa0b9311880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:08.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:29Z" 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:29Z" 2023-09-30T00:47:08.236 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:34.928+0000 7fc29fe10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:34.929+0000 7fc29fe10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:34Z" 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.243 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.245 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.246 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.247 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.248 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:34Z" 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:40.048+0000 7f8a07e24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:40.050+0000 7f8a07e24880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:40Z" 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.258 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:40Z" 2023-09-30T00:47:08.261 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:45.166+0000 7f423b80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:45.167+0000 7f423b80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:45Z" 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.269 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.271 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.272 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.272 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.272 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.272 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.273 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.274 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.275 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:45Z" 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:50.324+0000 7fdeb2213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:50.326+0000 7fdeb2213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:50Z" 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.283 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.285 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.286 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:50Z" 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:55.448+0000 7f48c7136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:46:55.449+0000 7f48c7136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:46:55Z" 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.297 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.299 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.300 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.301 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.302 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:46:55Z" 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:00.575+0000 7fe9648b8880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:00.576+0000 7fe9648b8880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:00Z" 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.311 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.003827_212.1", 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.003827Z", 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.316 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:00Z" 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.003827_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.003827Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.003827_212.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.003827_212.1 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:05.726+0000 7f1a2982e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:05.727+0000 7f1a2982e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026484.251433_236.1", 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:05Z" 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.324 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.325 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.326 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.327 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026484.251433_236.1", 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.328 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.329 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:05Z" 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.330 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.251433_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.251433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.251433_236.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.251433_236.1 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:10.836+0000 7fc295cb6880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:10.837+0000 7fc295cb6880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.123832_226.1", 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:10Z" 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.338 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.340 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.123832_226.1", 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-30T00:47:08.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:10Z" 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:15.961+0000 7f9ca6136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:15.962+0000 7f9ca6136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:15Z" 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.351 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.354 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.356 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:15Z" 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:21.092+0000 7f8596927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:21.092+0000 7f8596927880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:21Z" 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.364 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.365 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.366 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.369 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:21Z" 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:26.205+0000 7fdd1fa67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:26.206+0000 7fdd1fa67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:26Z" 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.377 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.382 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:26Z" 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:31.334+0000 7fcc4ed1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:31.341+0000 7fcc4ed1e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:31Z" 2023-09-30T00:47:08.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.390 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.392 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.393 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:31Z" 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:36.493+0000 7fbb53f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:36.493+0000 7fbb53f23880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:36Z" 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.402 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.407 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:36Z" 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:41.618+0000 7f6ecd626880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:41.619+0000 7f6ecd626880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.081380_219.1", 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:41Z" 2023-09-30T00:47:08.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.417 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.418 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.081380_219.1", 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.081380Z", 2023-09-30T00:47:08.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:41Z" 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.081380_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.081380Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.081380_219.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.081380_219.1 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:46.749+0000 7f84ead29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:46.750+0000 7f84ead29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:46Z" 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.428 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.432 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.433 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:46Z" 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:51.870+0000 7f33e3c9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:51.871+0000 7f33e3c9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:51Z" 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.442 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.444 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.445 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.448 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:51Z" 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:56.993+0000 7f9b4931e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:47:56.994+0000 7f9b4931e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:47:56Z" 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.456 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.458 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.462 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:47:56Z" 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:02.182+0000 7fb165815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:02.183+0000 7fb165815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:02Z" 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.469 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.009067_215.1", 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.009067Z", 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.474 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:02Z" 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.009067_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.009067Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.009067_215.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.009067_215.1 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:07.305+0000 7f1228ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:07.306+0000 7f1228ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:07Z" 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:07Z" 2023-09-30T00:47:08.486 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:12.423+0000 7f0b267a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:12.424+0000 7f0b267a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:12Z" 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.493 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.494 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.495 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.978724_210.1", 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.978724Z", 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.496 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.497 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:12Z" 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.978724_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.978724Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.978724_210.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.978724_210.1 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:17.618+0000 7f4625812880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:17.619+0000 7f4625812880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:17Z" 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.505 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.506 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.507 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.508 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.509 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:17Z" 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:22.725+0000 7fed9c9af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:22.725+0000 7fed9c9af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026484.251433_236.1", 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:22Z" 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.517 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.518 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.519 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026484.251433_236.1", 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:04.251433Z", 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.520 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.521 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:22Z" 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.251433_236.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.251433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.251433_236.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.251433_236.1 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:27.835+0000 7fcd4ad4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:27.836+0000 7fcd4ad4b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:27Z" 2023-09-30T00:47:08.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.529 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.530 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.965243_207.1", 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.965243Z", 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.532 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.533 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:27Z" 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.965243_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.965243Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.965243_207.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.965243_207.1 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:32.978+0000 7f1847929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:32.979+0000 7f1847929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:32Z" 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.540 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.541 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.977901_209.1", 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.977901Z", 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.545 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:32Z" 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.977901_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.977901Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.977901_209.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.977901_209.1 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:38.098+0000 7f7d82e9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:38.099+0000 7f7d82e9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006025_213.1", 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:38Z" 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006025_213.1", 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006025Z", 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.555 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:38Z" 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006025_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006025_213.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006025_213.1 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:43.221+0000 7efc17929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:43.222+0000 7efc17929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.095615_221.1", 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:43Z" 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.564 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.565 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.566 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.095615_221.1", 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.567 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.568 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:43Z" 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:48.361+0000 7f7d95737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:48.362+0000 7f7d95737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.026351_217.1", 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:48Z" 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.576 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.577 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.026351_217.1", 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.026351Z", 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.581 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:48Z" 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.026351_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.026351Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.026351_217.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.026351_217.1 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:53.476+0000 7f0d62726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:53.477+0000 7f0d62726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:53Z" 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.589 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.590 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.591 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.980766_211.1", 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.980766Z", 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.592 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.593 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:53Z" 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.980766_211.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.980766Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.980766_211.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.980766_211.1 2023-09-30T00:47:08.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:58.589+0000 7f0090133880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:48:58.590+0000 7f0090133880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:48:58Z" 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.601 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.602 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.603 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.604 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.605 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:48:58Z" 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:03.693+0000 7fdb69026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:03.694+0000 7fdb69026880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.127019_228.1", 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.127019Z", 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:03Z" 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.613 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.127019_228.1", 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.127019Z", 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.616 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.617 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:03Z" 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.127019_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.127019Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.127019_228.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.127019_228.1 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:08.803+0000 7f94f7537880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:08.804+0000 7f94f7537880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:08Z" 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.624 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.629 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:08Z" 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:13.908+0000 7f60a9a14880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:13.909+0000 7f60a9a14880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:13Z" 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.637 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.638 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.639 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.023985_216.1", 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.023985Z", 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.640 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.641 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:13Z" 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.023985_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.023985Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.023985_216.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.023985_216.1 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:19.020+0000 7f5afd619880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:19.021+0000 7f5afd619880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:19Z" 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.649 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.650 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026482.975764_208.1", 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:02.975764Z", 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:19Z" 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026482.975764_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:02.975764Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026482.975764_208.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026482.975764_208.1 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:24.129+0000 7f2765e26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:24.130+0000 7f2765e26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:24Z" 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.661 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.665 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.666 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.006853_214.1", 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.006853Z", 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.667 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.668 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:24Z" 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.006853_214.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.006853Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.006853_214.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.006853_214.1 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:29.239+0000 7f66b654b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:29.240+0000 7f66b654b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:29Z" 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.676 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.677 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:29Z" 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:34.364+0000 7fa059811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:34.364+0000 7fa059811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:34Z" 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.688 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.689 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.690 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.691 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.692 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:34Z" 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:39.536+0000 7f8d29c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:39.536+0000 7f8d29c10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:39Z" 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.704 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:39Z" 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:44.641+0000 7f954769d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:44.641+0000 7f954769d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:44Z" 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.712 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.713 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.714 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.715 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.716 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:44Z" 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-30T00:47:08.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:49.755+0000 7ff329c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:49.756+0000 7ff329c8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:49Z" 2023-09-30T00:47:08.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.110230_223.1", 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.110230Z", 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.728 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:49Z" 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.110230_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.110230Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.110230_223.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.110230_223.1 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:54.892+0000 7f26b65af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:49:54.893+0000 7f26b65af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:49:54Z" 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.735 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.736 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.079039_218.1", 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.079039Z", 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.740 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:49:54Z" 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.079039_218.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.079039Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.079039_218.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.079039_218.1 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:00.018+0000 7fd98849d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:00.019+0000 7fd98849d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.216959_231.1", 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.216959Z", 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:00Z" 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.747 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.749 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.216959_231.1", 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.216959Z", 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.751 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.752 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:00Z" 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.216959_231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.216959Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.216959_231.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.216959_231.1 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:05.126+0000 7f3dba329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:05.127+0000 7f3dba329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:05Z" 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.759 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.760 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:05Z" 2023-09-30T00:47:08.764 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:10.242+0000 7f6360813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:10.243+0000 7f6360813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.201655_230.1", 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.201655Z", 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:10Z" 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.772 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.201655_230.1", 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.201655Z", 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.774 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.775 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:10Z" 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.201655_230.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.201655Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.201655_230.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.201655_230.1 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:15.383+0000 7fea31844880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:15.384+0000 7fea31844880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.123832_226.1", 2023-09-30T00:47:08.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:15Z" 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.783 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.784 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.785 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.123832_226.1", 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.123832Z", 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.786 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.787 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:15Z" 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.123832_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.123832Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.123832_226.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.123832_226.1 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:20.548+0000 7fd7b5526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:20.551+0000 7fd7b5526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.095615_221.1", 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:20Z" 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.795 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.796 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.797 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.095615_221.1", 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.095615Z", 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.798 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.799 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:20Z" 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.095615_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.095615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.095615_221.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.095615_221.1 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:25.704+0000 7fe400123880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:25.705+0000 7fe400123880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026484.254025_237.1", 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:04.254025Z", 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:25Z" 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.807 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.808 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.809 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026484.254025_237.1", 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:04.254025Z", 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.810 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.811 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:25Z" 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026484.254025_237.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:04.254025Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026484.254025_237.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026484.254025_237.1 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:30.818+0000 7fb3b2929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:30.819+0000 7fb3b2929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.111037_224.1", 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:30Z" 2023-09-30T00:47:08.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.819 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.820 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.111037_224.1", 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.111037Z", 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.823 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:30Z" 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.111037_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.111037Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.111037_224.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.111037_224.1 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:35.931+0000 7f109a013880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:35.932+0000 7f109a013880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:35Z" 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.830 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.835 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:35Z" 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:41.048+0000 7f935bb31880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:41.049+0000 7f935bb31880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.113101_225.1", 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:41Z" 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.113101_225.1", 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.113101Z", 2023-09-30T00:47:08.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:41Z" 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.848 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.113101_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.113101Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.113101_225.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.113101_225.1 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:46.181+0000 7f54e8412880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:50:46.184+0000 7f54e8412880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:08.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:50:46Z" 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.855 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026483.093360_220.1", 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:28:03.093360Z", 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:50:46Z" 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.860 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696026483.093360_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:28:03.093360Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696026483.093360_220.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696027481.827984_326.1 target=1_1696026483.093360_220.1 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.861 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:08.862 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2023-09-30T00:47:08.862 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:08.864 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:08.864 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:08.864 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:08.864 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1148, in test_bucket_sync_disable 2023-09-30T00:47:08.864 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 375, in zonegroup_data_checkpoint 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 366, in zone_data_checkpoint 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests: assert False, 'failed data checkpoint for target_zone=%s source_zone=%s' % \ 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-52 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-52/ 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-52/ 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-52/ 2023-09-30T00:47:08.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:55:09 GMT 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:/dmuhsu-52/ 2023-09-30T00:47:08.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ggwFeLIc4npdYMUhMUsLeLjJwEc= 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ggwFeLIc4npdYMUhMUsLeLjJwEc=', 'Content-Length': '0'} 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ea01ca120eef615e-00651755cd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-53 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-53/ 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-53/ 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-53/ 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:08.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:55:09 GMT 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:/dmuhsu-53/ 2023-09-30T00:47:08.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:E9AKkpBtj7R1N4lWigBjuoddVV4= 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E9AKkpBtj7R1N4lWigBjuoddVV4=', 'Content-Length': '0'} 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007e049104c38943f3-00651755cd-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-54 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-54/ 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-54/ 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-54/ 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:08.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:55:09 GMT 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:/dmuhsu-54/ 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:MTrP4+ToJCvF4or7JhV87q51z6A= 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:09 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MTrP4+ToJCvF4or7JhV87q51z6A=', 'Content-Length': '0'} 2023-09-30T00:47:08.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000bb155adc81ade653-00651755cd-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:08.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028109.165504_437.1", 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:55:09.165504Z" 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:08.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:09.530+0000 7fe55500f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:09.531+0000 7fe55500f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:09Z" 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.880 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.881 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.882 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.884 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:09Z" 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.885 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:14.638+0000 7f9a5ecc0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:14.639+0000 7f9a5ecc0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:14Z" 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.892 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.893 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.894 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.895 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:14Z" 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.897 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:19.747+0000 7ff742a8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:19.747+0000 7ff742a8b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:19Z" 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.904 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:19Z" 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:24.854+0000 7f8e771af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:24.855+0000 7f8e771af880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:24Z" 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.920 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:24Z" 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:29.961+0000 7fddb0e99880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:29.962+0000 7fddb0e99880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:29Z" 2023-09-30T00:47:08.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.928 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.929 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.930 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.932 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:29Z" 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:35.070+0000 7fa98b4c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:35.071+0000 7fa98b4c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:35Z" 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.939 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.944 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:35Z" 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:40.179+0000 7fa1b6a19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:40.180+0000 7fa1b6a19880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:40Z" 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.953 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:40Z" 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:45.289+0000 7f9b5e00f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:45.290+0000 7f9b5e00f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:45Z" 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.964 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.965 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:45Z" 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:50.399+0000 7fee27224880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:50.400+0000 7fee27224880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:50Z" 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.975 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.976 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.977 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.978 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.979 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:50Z" 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:55.507+0000 7fab9534b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:55:55.508+0000 7fab9534b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:55:55Z" 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.987 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.988 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:08.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:55:55Z" 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:08.992 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:00.615+0000 7f8b65f29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:00.616+0000 7f8b65f29880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:08.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:08.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:08.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:08.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:00Z" 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:08.999 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.001 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:00Z" 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:05.724+0000 7fa98111c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:05.725+0000 7fa98111c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:05Z" 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.011 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.012 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:05Z" 2023-09-30T00:47:09.016 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:10.832+0000 7f814af1a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:10.833+0000 7f814af1a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:10Z" 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.023 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:10Z" 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:15.939+0000 7f3aa3921880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:15.940+0000 7f3aa3921880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:15Z" 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.035 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.036 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.037 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:15Z" 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:21.049+0000 7fab24133880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:21.050+0000 7fab24133880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:21Z" 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.048 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.049 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.050 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:21Z" 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:26.161+0000 7feb0e811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:26.161+0000 7feb0e811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:26Z" 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.059 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.060 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:26Z" 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:31.274+0000 7f32e8521880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:31.275+0000 7f32e8521880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:31Z" 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.072 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:31Z" 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:36.384+0000 7fe928a67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:36.385+0000 7fe928a67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:36Z" 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.083 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.084 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.085 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.086 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.087 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:36Z" 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:41.492+0000 7ffaf0736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:41.493+0000 7ffaf0736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:41Z" 2023-09-30T00:47:09.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.095 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.096 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.097 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027481.827984_326.1", 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:44:41.827984Z", 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.098 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.099 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:41Z" 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696027481.827984_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:44:41.827984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696027481.827984_326.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696027481.827984_326.1 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:46.603+0000 7f862a127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:46.604+0000 7f862a127880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:46Z" 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.111 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:46Z" 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:51.714+0000 7f18e6729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:51.715+0000 7f18e6729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:51Z" 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.120 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.121 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.122 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.123 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:51Z" 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:56.828+0000 7fc3f3e32880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:56:56.829+0000 7fc3f3e32880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:56:56Z" 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.130 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.133 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.135 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:56:56Z" 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:01.968+0000 7f2719212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:01.969+0000 7f2719212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.724469_388.1", 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:01Z" 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.143 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.144 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.145 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.724469_388.1", 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.146 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.147 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:01Z" 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.724469_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.724469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.724469_388.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.724469_388.1 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:07.071+0000 7f22d1526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:07.072+0000 7f22d1526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:07Z" 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.155 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.156 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.157 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.158 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.159 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:07Z" 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:12.192+0000 7f89e900f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:12.193+0000 7f89e900f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:12Z" 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.167 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.168 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.169 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.170 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.171 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:12Z" 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:17.294+0000 7fd9bba14880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:17.295+0000 7fd9bba14880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:17Z" 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.179 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.180 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.181 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.182 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.183 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:17Z" 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:22.405+0000 7fc147815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:22.407+0000 7fc147815880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.910421_403.1", 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:22Z" 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.910421_403.1", 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.195 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:22Z" 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.910421_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.910421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.910421_403.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.910421_403.1 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:27.554+0000 7f8c50524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:27.555+0000 7f8c50524880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.736877_389.1", 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.736877Z", 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:27Z" 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.202 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.736877_389.1", 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.736877Z", 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.207 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:27Z" 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.736877_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.736877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.736877_389.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.736877_389.1 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:32.656+0000 7fc0dcc10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:32.657+0000 7fc0dcc10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.839654_400.1", 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.839654Z", 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:32Z" 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.214 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.215 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.839654_400.1", 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.839654Z", 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.219 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:32Z" 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.839654_400.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.839654Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.839654_400.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.839654_400.1 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:37.768+0000 7ff4ffa9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:37.769+0000 7ff4ffa9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:37Z" 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.226 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:37Z" 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.231 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:42.874+0000 7fe823a0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:42.875+0000 7fe823a0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:42Z" 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.241 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.242 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:42Z" 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:47.981+0000 7f07b0213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:47.982+0000 7f07b0213880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:47Z" 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.251 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.252 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.253 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.254 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:47Z" 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:53.108+0000 7f04ca40f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:53.109+0000 7f04ca40f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:53Z" 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.262 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.264 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.265 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.266 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:53Z" 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:58.211+0000 7f654069d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:57:58.211+0000 7f654069d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:57:58Z" 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.274 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.275 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.276 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.823047_398.1", 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.823047Z", 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.278 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:57:58Z" 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.823047_398.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.823047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.823047_398.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.823047_398.1 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:03.334+0000 7f0377c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:03.337+0000 7f0377c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:03Z" 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.288 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.289 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.290 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.291 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.292 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:03Z" 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:08.464+0000 7f108eb75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:08.465+0000 7f108eb75880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.910421_403.1", 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:08Z" 2023-09-30T00:47:09.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.910421_403.1", 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.910421Z", 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.304 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:08Z" 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.910421_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.910421Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.910421_403.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.910421_403.1 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:13.572+0000 7f103d40f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:13.573+0000 7f103d40f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:13Z" 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.311 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.313 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.316 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:13Z" 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:18.673+0000 7fb52cf2a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:18.674+0000 7fb52cf2a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:18Z" 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.323 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.326 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.654382_385.1", 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.654382Z", 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.327 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.328 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:18Z" 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.654382_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.654382Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.654382_385.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.654382_385.1 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:23.773+0000 7fc080a26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:23.774+0000 7fc080a26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:23Z" 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.335 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.336 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.340 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:23Z" 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:28.885+0000 7f5ec380a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:28.887+0000 7f5ec380a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:28Z" 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.347 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.348 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.352 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:28Z" 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:34.051+0000 7efe7da99880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:34.052+0000 7efe7da99880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:34Z" 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.359 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.364 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:34Z" 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:39.158+0000 7f5b00e11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:39.159+0000 7f5b00e11880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:39Z" 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.371 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.642877_382.1", 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.642877Z", 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.375 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.376 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:39Z" 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.642877_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.642877Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.642877_382.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.642877_382.1 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:44.260+0000 7f302228b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:44.261+0000 7f302228b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.854141_401.1", 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:44Z" 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.383 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.384 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.854141_401.1", 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.386 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.388 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:44Z" 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.854141_401.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.854141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.854141_401.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.854141_401.1 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:49.366+0000 7ffb11a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:49.367+0000 7ffb11a9d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:49Z" 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.395 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.740786_390.1", 2023-09-30T00:47:09.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.740786Z", 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.400 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:49Z" 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.740786_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.740786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.740786_390.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.740786_390.1 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:54.480+0000 7faf64467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:54.481+0000 7faf64467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:54Z" 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.407 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.408 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.409 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.412 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:54Z" 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:59.585+0000 7f98e2244880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:58:59.586+0000 7f98e2244880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:58:59Z" 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.419 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.420 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.899474_402.1", 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.899474Z", 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:58:59Z" 2023-09-30T00:47:09.424 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.899474_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.899474Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.899474_402.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.899474_402.1 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:04.697+0000 7f22ec328880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:04.699+0000 7f22ec328880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.854141_401.1", 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:04Z" 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.854141_401.1", 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.854141Z", 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:04Z" 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.436 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.854141_401.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.854141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.854141_401.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.854141_401.1 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:09.829+0000 7fb4be2bc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:09.830+0000 7fb4be2bc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028106.724469_388.1", 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:09Z" 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028106.724469_388.1", 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:06.724469Z", 2023-09-30T00:47:09.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.447 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:09Z" 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028106.724469_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.724469Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028106.724469_388.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028109.165504_437.1 target=1_1696028106.724469_388.1 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:14.940+0000 7f887f017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T22:59:14.941+0000 7f887f017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028109.165504_437.1", 2023-09-30T00:47:09.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:09.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:14Z" 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.455 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.456 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.457 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028109.165504_437.1", 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.458 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.459 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:14Z" 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028109.165504_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:09.165504Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:59:15.280+0000 7fef43415880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T22:59:15.281+0000 7fef43415880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:09.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:09.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:09.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028109.165504_437.1", 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:09.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T22:59:15Z" 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.467 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.468 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.469 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028109.165504_437.1", 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:55:09.165504Z", 2023-09-30T00:47:09.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:09.471 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:15Z" 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028109.165504_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:09.165504Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028109.165504_437.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-52', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-52', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-52 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-53', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-53', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-53 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dmuhsu-54', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dmuhsu-54', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dmuhsu-54 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.29", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000078", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.history", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.23", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000070", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000104", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000026", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.8", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000028", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:3", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.30", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000040", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.6", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000015", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000069", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000095", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000003", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000047", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000035", 2023-09-30T00:47:09.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000037", 2023-09-30T00:47:09.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000024", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000000", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000031", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.3", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:0", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_loggenerations_metadata", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000076", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000113", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000005", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000011", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000058", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000052", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.21", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000088", 2023-09-30T00:47:09.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000080", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000090", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000110", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000096", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000087", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000008", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000006", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000029", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000089", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000022", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:1", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bilog.trim", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000016", 2023-09-30T00:47:09.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000041", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000018", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000092", 2023-09-30T00:47:09.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.27", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.4", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000014", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000112", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000007", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000021", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.16", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000064", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.2.w7s8+TIelzD.0", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000071", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000074", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.0.QYJY6t9SUVl.0", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000081", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000009", 2023-09-30T00:47:09.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000121", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.1", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.17", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000125", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000082", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000105", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000059", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000077", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000032", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.3.4KVdP9WnyOm.0", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.24", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000053", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000091", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1.retry", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000065", 2023-09-30T00:47:09.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000083", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000010", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.7", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.3", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000045", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000002", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.3", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.12", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000116", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:6", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.25", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000034", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000101", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.13", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000079", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000049", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000117", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000044", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.14", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000066", 2023-09-30T00:47:09.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000068", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000085", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000073", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000038", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.10", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000118", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000036", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000103", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000119", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1.retry", 2023-09-30T00:47:09.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000098", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000027", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000019", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.3", 2023-09-30T00:47:09.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.1.V9YRRUmTpg1.0", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000039", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2.retry", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2.retry", 2023-09-30T00:47:09.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000100", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000093", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0.retry", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000004", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000063", 2023-09-30T00:47:09.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000122", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000057", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0.retry", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000054", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000114", 2023-09-30T00:47:09.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.2", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:2", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.0", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000012", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000084", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000043", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:4", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.11", 2023-09-30T00:47:09.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000111", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000048", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000020", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000099", 2023-09-30T00:47:09.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000056", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.19", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.28", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000072", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000062", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:9", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000115", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.1", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000033", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000025", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000123", 2023-09-30T00:47:09.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.18", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000108", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1", 2023-09-30T00:47:09.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000094", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000060", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000109", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.15", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000013", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000051", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000106", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000017", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000023", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000107", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000046", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.26", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.22", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.2", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000067", 2023-09-30T00:47:09.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.9", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000086", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.5", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.20", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000050", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000102", 2023-09-30T00:47:09.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:7", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000124", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000001", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0", 2023-09-30T00:47:09.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000126", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:8", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000055", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000061", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:10", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000097", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:5", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000042", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000075", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000120", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000030", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9" 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: [ 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000078", 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "meta.history", 2023-09-30T00:47:09.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000070", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000104", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000026", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.8", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000028", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000040", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2", 2023-09-30T00:47:09.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.6", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000015", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000069", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000095", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000003", 2023-09-30T00:47:09.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000047", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000035", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000037", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000024", 2023-09-30T00:47:09.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000000", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000031", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.3", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_loggenerations_metadata", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000076", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000113", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000005", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000011", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000058", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000052", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000088", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000080", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000090", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000110", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000096", 2023-09-30T00:47:09.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.3", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000087", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000008", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0.retry", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000006", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000029", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000089", 2023-09-30T00:47:09.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000022", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bilog.trim", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000016", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000041", 2023-09-30T00:47:09.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000018", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000092", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.4", 2023-09-30T00:47:09.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000014", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000112", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000007", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000021", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000064", 2023-09-30T00:47:09.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000071", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000074", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000081", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000009", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000121", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.1", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000125", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1.retry", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000082", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000105", 2023-09-30T00:47:09.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000059", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000077", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000032", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000053", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000091", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000065", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2", 2023-09-30T00:47:09.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000083", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000010", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.7", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.3", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "mdlog.sync-status.shard.2", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000045", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000002", 2023-09-30T00:47:09.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000116", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000034", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000101", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "mdlog.sync-status.shard.1", 2023-09-30T00:47:09.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000079", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000049", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000117", 2023-09-30T00:47:09.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000044", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000066", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000068", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000085", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000073", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000038", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.10", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000118", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000036", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000103", 2023-09-30T00:47:09.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "mdlog.sync-status.shard.3", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000119", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1.retry", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000098", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "mdlog.sync-status.shard.0", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000027", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "mdlog.sync-status", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000019", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.3", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000039", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.2.retry", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000100", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000093", 2023-09-30T00:47:09.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0.retry", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000004", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000063", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000122", 2023-09-30T00:47:09.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000057", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000054", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000114", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.2", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.0", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.1", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000012", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000084", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000043", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.11", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.0.85QjD62xz2M.0", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000111", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000048", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000020", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000099", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000056", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000072", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000062", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000115", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.1", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000033", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000025", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000123", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000108", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000094", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000060", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000109", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000013", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000051", 2023-09-30T00:47:09.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000106", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000017", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000023", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000107", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000046", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.2", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000067", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.1.gEURCYoqqI1.0", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.9", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000086", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync.error-log.5", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "data_log.2.3y0gFtudtMP.0", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000050", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000102", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2.retry", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000124", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000001", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "datalog.sync-status.shard.4e047d8a-7fd9-483a-b356-0bddc8ecae27.0", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-30T00:47:09.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000126", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000055", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000061", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000097", 2023-09-30T00:47:09.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000042", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000075", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000120", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "obj_delete_at_hint.0000000030", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.4e047d8a-7fd9-483a-b356-0bddc8ecae27:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8" 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000078", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.history", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000070", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000104", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.2", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000026", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.8", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000028", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:0", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000040", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.6", 2023-09-30T00:47:09.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000015", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000069", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000095", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000003", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000047", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:7", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000035", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:2", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000037", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:6", 2023-09-30T00:47:09.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:2", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.2.jkVBsvUHjGr.0", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000024", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000000", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000031", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:1", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.3", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:5", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:4", 2023-09-30T00:47:09.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:8", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:5", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_loggenerations_metadata", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:5", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000076", 2023-09-30T00:47:09.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000113", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000005", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000011", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000058", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000052", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:7", 2023-09-30T00:47:09.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000088", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000080", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000090", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000110", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:0", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.0", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000096", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.3", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000087", 2023-09-30T00:47:09.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000008", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0.retry", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000006", 2023-09-30T00:47:09.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:10", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000029", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000089", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000022", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:5", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:10", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bilog.trim", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:9", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000016", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:3", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000041", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:10", 2023-09-30T00:47:09.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:8", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000018", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000092", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.4", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000014", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:6", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000112", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:9", 2023-09-30T00:47:09.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000007", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000021", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:3", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:1", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000064", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000071", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000074", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:0", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:7", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:3", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:8", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000081", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000009", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000121", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.1", 2023-09-30T00:47:09.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:1", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000125", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1.retry", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000082", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:7", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:7", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:1", 2023-09-30T00:47:09.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000105", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000059", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:9", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000077", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000032", 2023-09-30T00:47:09.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:8", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:4", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:7", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000053", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000091", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:6", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1.retry", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000065", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:1", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000083", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000010", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:9", 2023-09-30T00:47:09.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:10", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.1.i6/TT7IApej.0", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.7", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.3", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:0", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:3", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:3", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:8", 2023-09-30T00:47:09.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:8", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mdlog.sync-status.shard.2", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:2", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:6", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:5", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000045", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000002", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:9", 2023-09-30T00:47:09.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.3", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000116", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:5", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:0", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:6", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:6", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000034", 2023-09-30T00:47:09.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:9", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:3", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:8", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000101", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:7", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mdlog.sync-status.shard.1", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:10", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:6", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:4", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000079", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000049", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000117", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000044", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:3", 2023-09-30T00:47:09.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:1", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000066", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000068", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:7", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000085", 2023-09-30T00:47:09.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:2", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:7", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:6", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000073", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:9", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000038", 2023-09-30T00:47:09.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.10", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000118", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:4", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000036", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:5", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000103", 2023-09-30T00:47:09.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:9", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mdlog.sync-status.shard.3", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:6", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:5", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:9", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000119", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000098", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:5", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:2", 2023-09-30T00:47:09.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:0", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mdlog.sync-status.shard.0", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000027", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:2", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mdlog.sync-status", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000019", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:5", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:5", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000039", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:3", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:0", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.2.retry", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:3", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:1", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:7", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:3", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44", 2023-09-30T00:47:09.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:7", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:3", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:2", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000100", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:0", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:7", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:9", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:8", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:10", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000093", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:2", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:7", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000004", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000063", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000122", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:9", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000057", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:7", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.0.retry", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000054", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000114", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:3", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:9", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:7", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.2", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.0", 2023-09-30T00:47:09.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:1", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000012", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:1", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "meta.log.6a43d71a-d207-46f3-bdd8-e7a78954a184.3", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:0", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000084", 2023-09-30T00:47:09.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000043", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:10", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:10", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:10", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:4", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:3", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:4", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:9", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:0", 2023-09-30T00:47:09.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:4", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:9", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000111", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000048", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:10", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:6", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:4", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:10", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000020", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:9", 2023-09-30T00:47:09.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.1", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000099", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000056", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:5", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:0", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000072", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:10", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000062", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:9", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000115", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.1", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000033", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:4", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:2", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:5", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:0", 2023-09-30T00:47:09.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:6", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000025", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:2", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:6", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000123", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:4", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:10", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:3", 2023-09-30T00:47:09.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000108", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:4", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:6", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.e97e2b17-58fb-4667-bf95-8d29031e4378.1", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000094", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:1", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000060", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000109", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43", 2023-09-30T00:47:09.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:6", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:8", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000013", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:9", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:0", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.0", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:1", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:4", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:8", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:8", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000051", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000106", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:9", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000017", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:1", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:3", 2023-09-30T00:47:09.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:2", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:10", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:8", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:10", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000023", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:8", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:5", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:6", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:2", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:8", 2023-09-30T00:47:09.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:2", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000107", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:1", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000046", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:4", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:4", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:8", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:6", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:0", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:3", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:7", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:10", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:4", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:5", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:0", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.2", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:8", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:8", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000067", 2023-09-30T00:47:09.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:9", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:7", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:0", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:0", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.9", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000086", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:5", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:8", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync.error-log.5", 2023-09-30T00:47:09.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-11:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.11:2", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:3", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:1", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:7", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:5", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:10", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-12:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.12:6", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:3", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:0", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-17:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.17:2", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:10", 2023-09-30T00:47:09.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:2", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:6", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:5", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:1", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:1", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:1", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:4", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-24:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.24:3", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:1", 2023-09-30T00:47:09.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:2", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:2", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000050", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000102", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "datalog.sync-status.shard.d5be89e5-4ad9-4a27-8129-17eff73a6a3e.2.retry", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:8", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:10", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13:5", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22:1", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:7", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:4", 2023-09-30T00:47:09.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000124", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:9", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000001", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-43:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.43:5", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:5", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:2", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_log.0.zEAIIZaW88R.0", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000126", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:6", 2023-09-30T00:47:09.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:4", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-18:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.18:4", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-14:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.14:10", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000055", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-44:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.44:3", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:10", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:8", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23:7", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000061", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-19:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.19:6", 2023-09-30T00:47:09.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:4", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000097", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:2", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-10:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.10:3", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-16:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.16:4", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:6", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000042", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-26:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.26:0", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-45:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.6847.1:0", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-15:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.15:1", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-23:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.23", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:0", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-22:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.22", 2023-09-30T00:47:09.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-21:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.21:7", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000075", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000120", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj_delete_at_hint.0000000030", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:2", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-20:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.20:1", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.full-sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-13:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.13", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.d5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-25:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.25:8", 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket.sync-status.e97e2b17-58fb-4667-bf95-8d29031e4378:dmuhsu-42:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.4358.42:9" 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: 2023-09-30T00:47:09.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data checkpoint: source=test-zone1 target=test-zone2 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000002904", 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:59:15.714290Z" 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000005580", 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:59:15.715237Z" 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000006707", 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:59:15.716232Z" 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000001634", 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:59:15.718951Z" 2023-09-30T00:47:09.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:16Z" 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.656 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.657 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:16Z" 2023-09-30T00:47:09.661 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:21Z" 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.667 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.668 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.671 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:21Z" 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.672 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:26Z" 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.679 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.680 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.683 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.685 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:26Z" 2023-09-30T00:47:09.685 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.685 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:31Z" 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.692 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:31Z" 2023-09-30T00:47:09.696 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:36Z" 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.703 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:36Z" 2023-09-30T00:47:09.707 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:42Z" 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.714 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.716 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.717 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.718 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:42Z" 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:47Z" 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.725 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.726 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.727 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.728 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.729 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:47Z" 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:52Z" 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.736 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.740 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:52Z" 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T22:59:57Z" 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.747 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.749 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.750 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.751 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:59:57Z" 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:02Z" 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.758 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:02Z" 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:07Z" 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.769 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.771 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:07Z" 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:12Z" 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.780 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.782 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.783 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:12Z" 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:17Z" 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.791 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.794 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.795 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:17Z" 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:22Z" 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.802 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.803 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.804 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.806 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:22Z" 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:28Z" 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.813 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.814 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.815 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.816 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.817 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:28Z" 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:33Z" 2023-09-30T00:47:09.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.827 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:33Z" 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:38Z" 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.837 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.839 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.840 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.841 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:38Z" 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:43Z" 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.848 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.851 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.852 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:43Z" 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:48Z" 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.859 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.863 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:48Z" 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.864 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:53Z" 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.870 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:53Z" 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.875 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:00:58Z" 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.882 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.886 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:00:58Z" 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:03Z" 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.893 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.897 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:03Z" 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:08Z" 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:08Z" 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:14Z" 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.915 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.916 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.917 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.919 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:14Z" 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:19Z" 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.926 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.927 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.928 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.929 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.930 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:19Z" 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:24Z" 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.937 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.938 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.939 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.940 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:24Z" 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:29Z" 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.950 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.951 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.952 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:29Z" 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:34Z" 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.959 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.961 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.963 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:34Z" 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:39Z" 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.970 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.973 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:39Z" 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:44Z" 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.983 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.984 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.985 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:44Z" 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:49Z" 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:09.992 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:09.994 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.995 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:09.996 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:49Z" 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:09.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:09.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:09.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:01:55Z" 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.003 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.004 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.005 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.006 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.007 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:01:55Z" 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.008 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:00Z" 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.014 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.017 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.018 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.019 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:00Z" 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:05Z" 2023-09-30T00:47:10.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.026 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.027 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:05Z" 2023-09-30T00:47:10.030 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:10Z" 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.037 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:10Z" 2023-09-30T00:47:10.041 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:15Z" 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:15Z" 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:20Z" 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.058 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.060 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.061 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:20Z" 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:25Z" 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.069 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.070 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.071 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.072 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.073 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:25Z" 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.074 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:30Z" 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.080 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.082 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.083 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.084 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:30Z" 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:35Z" 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.091 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.097 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:35Z" 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:41Z" 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.104 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.105 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.107 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.108 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:41Z" 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:46Z" 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.115 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.116 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.117 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.119 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:46Z" 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:51Z" 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.126 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.127 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.128 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.130 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:51Z" 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:02:56Z" 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.137 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.140 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.141 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:56Z" 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:01Z" 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.148 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.149 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:01Z" 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.153 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:06Z" 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.160 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.161 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.162 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.163 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.164 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:06Z" 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:11Z" 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.171 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.173 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.174 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.175 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:11Z" 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:16Z" 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.182 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.185 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.186 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:16Z" 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:21Z" 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.193 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.195 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.196 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.197 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:21Z" 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:27Z" 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.204 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.205 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.206 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.207 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.208 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:27Z" 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:32Z" 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.215 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.217 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.218 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:32Z" 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:37Z" 2023-09-30T00:47:10.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.226 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.227 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:37Z" 2023-09-30T00:47:10.230 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:42Z" 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.236 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.240 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:42Z" 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:47Z" 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.247 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.249 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.251 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:47Z" 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:52Z" 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.258 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.259 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.260 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.261 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.262 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:52Z" 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:03:57Z" 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.269 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.270 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.271 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.272 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.273 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:03:57Z" 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:02Z" 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.280 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.282 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.283 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:02Z" 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:07Z" 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.291 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.293 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.294 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.295 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:07Z" 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:13Z" 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.307 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:13Z" 2023-09-30T00:47:10.311 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 11609692693196001247 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000" 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 0, 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 0 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:18Z" 2023-09-30T00:47:10.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "instance_id": 11609692693196001247 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.318 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002063", 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:16:54.204956Z" 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000004319", 2023-09-30T00:47:10.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:38:40.088667Z" 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000005306", 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:20:34.216869Z" 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000" 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: "total": 0, 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: "complete": 0 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:18Z" 2023-09-30T00:47:10.322 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:16:54.204956Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:38:40.088667Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:20:34.216869Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000002904', 1: '00000000000000000000:00000000000000005580', 2: '00000000000000000000:00000000000000006707', 3: '00000000000000000000:00000000000000001634'} 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000002063', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000005306', 3: ''} 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002904 target=00000000000000000000:00000000000000002063, shard=1 master=00000000000000000000:00000000000000005580 target=00000000000000000000:00000000000000004319, shard=2 master=00000000000000000000:00000000000000006707 target=00000000000000000000:00000000000000005306, shard=3 master=00000000000000000000:00000000000000001634 target= 2023-09-30T00:47:10.323 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.324 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1167, in test_bucket_sync_enable_right_after_disable 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-55 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/ 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/ 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/ 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/ 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:oWpMQjz3WjmBDR6QGff4m7UkLck= 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oWpMQjz3WjmBDR6QGff4m7UkLck=', 'Content-Length': '0'} 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b7bf093cdef5ac4c-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-56 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/ 2023-09-30T00:47:10.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/ 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/ 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.328 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/ 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:vRa6hmAPEtGu5SaFzj1v2dPeQMM= 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vRa6hmAPEtGu5SaFzj1v2dPeQMM=', 'Content-Length': '0'} 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e2a0d56bcfb76e2b-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-57 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/ 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/ 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/ 2023-09-30T00:47:10.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.330 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/ 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:l3ijMzIi/WMWtmJdnh7rEbHEPZM= 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:l3ijMzIi/WMWtmJdnh7rEbHEPZM=', 'Content-Length': '0'} 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ca4e7c5424524c48-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/ 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/ 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/ 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/ 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-30T00:47:10.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f7d6bd2886fef02-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/obj1 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/obj1 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/obj1 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/obj1 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:eIWWFkB/lvUOyp3IWm0YD+w+/Es= 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eIWWFkB/lvUOyp3IWm0YD+w+/Es='} 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e2e039585245362-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/ 2023-09-30T00:47:10.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/ 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/ 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.335 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/ 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089668def3248ac90-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/obj2 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/obj2 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/obj2 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/obj2 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.337 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3WKykTwxNpN8ntJoltAAenJGpQA= 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3WKykTwxNpN8ntJoltAAenJGpQA='} 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b70b4a2db03bc43e-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/ 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/ 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/ 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/ 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-30T00:47:10.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1d3a07bf24f3f8f-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/obj3 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/obj3 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/obj3 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.340 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/obj3 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:PxqPUPrShGQSHab4UK4QJsxHEwY= 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PxqPUPrShGQSHab4UK4QJsxHEwY='} 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008965d169d199dd78-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/ 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/ 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/ 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.342 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/ 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y= 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:arpLHgE5sk99tCpIzx9PS1zdG5Y=', 'Content-Length': '0'} 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fac7cf36bce219e-00651757f7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-55/obj4 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-55/obj4 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-55/obj4 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:/dmuhsu-55/obj4 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:epm8eyr3aibbvrnPIysEhH2Er/s= 2023-09-30T00:47:10.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:epm8eyr3aibbvrnPIysEhH2Er/s='} 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d52aa9ce76d067-00651757f7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/ 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/ 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/ 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/ 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027fd35f6a1cf6911-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/obj1 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/obj1 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/obj1 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.347 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/obj1 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:MjSuwqCv12JxOv2qjpSNtgmfn4Q= 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MjSuwqCv12JxOv2qjpSNtgmfn4Q='} 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001258b31831496167-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/ 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/ 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/ 2023-09-30T00:47:10.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.349 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/ 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017fd30f3ff3d5cc9-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/obj2 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/obj2 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/obj2 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/obj2 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:nqvrOzIxkKuxYjYN7GIbdy8X+kc= 2023-09-30T00:47:10.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nqvrOzIxkKuxYjYN7GIbdy8X+kc='} 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e428b19b46a963f8-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/ 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/ 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/ 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/ 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc8917c1c6cc44b8-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/obj3 2023-09-30T00:47:10.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/obj3 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/obj3 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.354 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/obj3 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1P/+rlG7FRRBclfDV526hczXhbk= 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1P/+rlG7FRRBclfDV526hczXhbk='} 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000075c1931c183e601-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/ 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/ 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/ 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/ 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.356 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU= 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YsH+FyZtYrAtPqnQcPNYYR4BBxU=', 'Content-Length': '0'} 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001379f075074cfeb-00651757f7-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-56/obj4 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-56/obj4 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-56/obj4 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:/dmuhsu-56/obj4 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Guc5d00QDtJDWLvtau2xWtuQP9M= 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Guc5d00QDtJDWLvtau2xWtuQP9M='} 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff1ce93489b56023-00651757f7-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/ 2023-09-30T00:47:10.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/ 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/ 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.359 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/ 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000400facd989fc8bad-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/obj1 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/obj1 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/obj1 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/obj1 2023-09-30T00:47:10.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1bk2icKiGo7klPNnzKQKT4aXm3I= 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1bk2icKiGo7klPNnzKQKT4aXm3I='} 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d19c18f5f7e8809a-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/ 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/ 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/ 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/ 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-30T00:47:10.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000463b97d8cbeefcb7-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/obj2 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/obj2 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/obj2 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.364 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/obj2 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:CdWQx28KODYEdD7PGzeQdEmgEz8= 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CdWQx28KODYEdD7PGzeQdEmgEz8='} 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010b15d5a160a5cc9-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/ 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/ 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/ 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.366 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/ 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7b25b4afe492201-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/obj3 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/obj3 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/obj3 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/obj3 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WeIMA3Im49/IEdE6GROw8Z548qI= 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WeIMA3Im49/IEdE6GROw8Z548qI='} 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b1dd4fe7aa0274e-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/ 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/ 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/ 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.369 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/ 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ= 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QO/hwsC4+nflvr7mnr1TkgN50uQ=', 'Content-Length': '0'} 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b1e922a3bd16bb-00651757f7-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-57/obj4 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-57/obj4 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-57/obj4 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:04:23 GMT 2023-09-30T00:47:10.371 INFO:tasks.rgw_multisite_tests:/dmuhsu-57/obj4 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:4bcDD8WWQJ1aEtKJUHOQSsrdHMI= 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:04:23 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4bcDD8WWQJ1aEtKJUHOQSsrdHMI='} 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f321d0151544b098-00651757f7-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:04:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028663.327363_455.1", 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:04:23.327363Z" 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:04:23.841+0000 7fb3ba721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:04:23.842+0000 7fb3ba721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028663.320433_452.1", 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:04:23.320433Z", 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:23Z" 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028663.320433_452.1", 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:04:23.320433Z", 2023-09-30T00:47:10.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.384 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:23Z" 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.320433_452.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.320433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.320433_452.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028663.327363_455.1 target=1_1696028663.320433_452.1 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:04:28.963+0000 7f0b67617880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:04:28.964+0000 7f0b67617880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028663.327363_455.1", 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:04:28Z" 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028663.327363_455.1", 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-30T00:47:10.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:28Z" 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.327363_455.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.327363Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:29.105+0000 7fca60d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:29.106+0000 7fca60d26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028663.327363_455.1", 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:04:29Z" 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.405 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028663.327363_455.1", 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:04:23.327363Z", 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.408 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:04:29Z" 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028663.327363_455.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:04:23.327363Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028663.327363_455.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-55 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-55 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-55 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:29.347+0000 7f108a60f880 1 bucket sync caught up with source: 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [00000000001.16.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.10.6, , , ] 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [00000000001.16.6, , 00000000001.23.6, , 00000000001.33.6, , , 00000000001.10.6, , , ] 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:29.347+0000 7f108a60f880 0 bucket checkpoint complete 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-55 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-55', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-55 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:29.448+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:34.448+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:39.449+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:44.450+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:49.450+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:54.451+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:04:59.451+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:04.452+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:09.452+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:14.453+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:19.454+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:24.453+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:29.454+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:34.455+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:39.456+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:44.457+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:49.457+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:54.458+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:05:59.458+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:04.459+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:09.459+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:14.460+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:19.461+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:24.460+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:29.461+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:34.462+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:39.463+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:44.464+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:49.464+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:54.465+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:06:59.465+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:04.466+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:09.466+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:14.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:19.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:24.467+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:29.468+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:34.468+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:39.469+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:44.470+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:49.470+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:54.471+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:07:59.472+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:04.472+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:09.473+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:14.474+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:19.473+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:24.474+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:29.475+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:34.475+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:39.476+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:44.477+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:49.478+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:54.478+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:08:59.479+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:04.479+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:09.480+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:14.481+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:19.480+0000 7ff642a44880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:24.481+0000 7ff642a44880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:24.481+0000 7ff642a44880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:24.481+0000 7ff642a44880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.417 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1202, in test_bucket_sync_disable_enable 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-58 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/ 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/ 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/ 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/ 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:0KTj9qj1iAGxY2pWXbOt27tAIWk= 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0KTj9qj1iAGxY2pWXbOt27tAIWk=', 'Content-Length': '0'} 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fa249fc20c5a46cb-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-59 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/ 2023-09-30T00:47:10.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/ 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/ 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.421 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/ 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x+UcKfAkblGyurBG8ohzlo6tqXc= 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x+UcKfAkblGyurBG8ohzlo6tqXc=', 'Content-Length': '0'} 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b45c1aee446a9aa2-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-60 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/ 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/ 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/ 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.423 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/ 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mWu6sbyd0EsL4FrtWHy6RL/gAEo= 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mWu6sbyd0EsL4FrtWHy6RL/gAEo=', 'Content-Length': '0'} 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b6f5652475b3b2e8-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/ 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/ 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/ 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/ 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-30T00:47:10.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078a3afe3a25447e5-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/obj1 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/obj1 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/obj1 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.426 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/obj1 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:r3eSGaBXvaxcCqSu3kjadoxhMKY= 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r3eSGaBXvaxcCqSu3kjadoxhMKY='} 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aeaf2e5c237aecf-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/ 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/ 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/ 2023-09-30T00:47:10.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/ 2023-09-30T00:47:10.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001643d5fb261df2e-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/obj2 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/obj2 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/obj2 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/obj2 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:a8GLC1smcbGcWBSFC73ir8IS+ao= 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a8GLC1smcbGcWBSFC73ir8IS+ao='} 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a31d48e522d1bae-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/ 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/ 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/ 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.431 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/ 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e99cd5e0c8f270b-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/obj3 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/obj3 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/obj3 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/obj3 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.433 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:RvQNOm0M7YgjsokJStZToDKPB5E= 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RvQNOm0M7YgjsokJStZToDKPB5E='} 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036b059868cca6fc5-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/ 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/ 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/ 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/ 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo= 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mZAebS7b8qVtOSrZGCDssYJmWgo=', 'Content-Length': '0'} 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a930750f33eb3e11-0065175924-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-58/obj4 2023-09-30T00:47:10.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-58/obj4 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-58/obj4 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:/dmuhsu-58/obj4 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zrQoELkuZ86CrPILwEDPQAYSrsE= 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zrQoELkuZ86CrPILwEDPQAYSrsE='} 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f52c31c77c4347c9-0065175924-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/ 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/ 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/ 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/ 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.438 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000975f2911f3887c03-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/obj1 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/obj1 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/obj1 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/obj1 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:CkcqC3FgC1E3RYvIQDlLWcauQp8= 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CkcqC3FgC1E3RYvIQDlLWcauQp8='} 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eb515244a2b9443-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/ 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/ 2023-09-30T00:47:10.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/ 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.441 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/ 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc43aaf64c32230-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/obj2 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/obj2 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/obj2 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/obj2 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tOaec1XFw1e4xWc9jMPGXCVg/iE= 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tOaec1XFw1e4xWc9jMPGXCVg/iE='} 2023-09-30T00:47:10.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041cd03506df62f36-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/ 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/ 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/ 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.444 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/ 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000226bd23c1fc01ff2-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/obj3 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/obj3 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/obj3 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.446 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/obj3 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:LXY8CWB7PlIo8j/ZyR3icLTEyjU= 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LXY8CWB7PlIo8j/ZyR3icLTEyjU='} 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010bf379230b61907-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/ 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/ 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/ 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/ 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc= 2023-09-30T00:47:10.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:s2phy4La7t0dlMwOPF1SPl0d2Sc=', 'Content-Length': '0'} 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad4c8b404398092-0065175924-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-59/obj4 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-59/obj4 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-59/obj4 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:/dmuhsu-59/obj4 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:b4VPeuxXjfryWjp5qadzlwPo/Uc= 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:b4VPeuxXjfryWjp5qadzlwPo/Uc='} 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084b7f2aafe5bfa1b-0065175924-14ab-test-zone2'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/ 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/ 2023-09-30T00:47:10.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/ 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.451 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/ 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f6b8fa2ff99462d-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/obj1 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/obj1 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/obj1 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/obj1 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NU4ilN00NC9XqjFMn5k/VU8tL0Y= 2023-09-30T00:47:10.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NU4ilN00NC9XqjFMn5k/VU8tL0Y='} 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046b956d5c903a95a-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/ 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/ 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/ 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/ 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fa3614f60078a3c-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/obj2 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/obj2 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/obj2 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.456 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/obj2 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:nAsjm3ICjOcXOru7TMbPr/xzOFY= 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nAsjm3ICjOcXOru7TMbPr/xzOFY='} 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c96be487ef250668-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/ 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/ 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/ 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/ 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-30T00:47:10.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c710f255370593a-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/obj3 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/obj3 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/obj3 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.459 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/obj3 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hjJFs7V2TqGg/0ovR+YoJvlcCQ0= 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hjJFs7V2TqGg/0ovR+YoJvlcCQ0='} 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e650479d27ddc8c2-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/ 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/ 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/ 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.461 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/ 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs= 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qrOG/DK8NvmIlxsWinOzdT6FWhs=', 'Content-Length': '0'} 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9fc81435926aa28-0065175924-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-60/obj4 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-60/obj4 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-60/obj4 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:10.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:09:24 GMT 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:/dmuhsu-60/obj4 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+QPrtg24KIPAGnxwp2M8314l244= 2023-09-30T00:47:10.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:09:24 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+QPrtg24KIPAGnxwp2M8314l244='} 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c3997396109eac9-0065175924-1ea0-test-zone3'), ('Date', 'Fri, 29 Sep 2023 23:09:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028964.627375_480.1", 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:09:24.627375Z" 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:09:25.106+0000 7f41cd728880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:09:25.107+0000 7f41cd728880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028964.621572_479.1", 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:09:24.621572Z", 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:09:25Z" 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028964.621572_479.1", 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:09:24.621572Z", 2023-09-30T00:47:10.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:09:25Z" 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.621572_479.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.621572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.621572_479.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696028964.627375_480.1 target=1_1696028964.621572_479.1 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:09:30.216+0000 7f72d4322880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:09:30.217+0000 7f72d4322880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696028964.627375_480.1", 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:09:30Z" 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.484 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.485 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.486 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028964.627375_480.1", 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.487 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.488 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:09:30Z" 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.627375_480.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.627375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:30.328+0000 7f9d1e327880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:30.329+0000 7f9d1e327880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696028964.627375_480.1", 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:09:30Z" 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.496 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.497 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.498 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028964.627375_480.1", 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:09:24.627375Z", 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.499 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.500 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:09:30Z" 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696028964.627375_480.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:09:24.627375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696028964.627375_480.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-58 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-58 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-58 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:30.560+0000 7ff22809d880 1 bucket sync caught up with source: 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [00000000001.24.6, , 00000000001.33.6, , 00000000001.25.6, , , 00000000001.28.6, , , ] 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [00000000001.24.6, , 00000000001.33.6, , 00000000001.25.6, , , 00000000001.28.6, , , ] 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:30.560+0000 7ff22809d880 0 bucket checkpoint complete 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-58 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-58 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:30.680+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:35.682+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:40.682+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:45.683+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:50.684+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:09:55.684+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:00.685+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:05.685+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:10.686+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:15.687+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:20.686+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:25.687+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:30.688+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:35.689+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:40.690+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:45.690+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:50.691+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:10:55.691+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:00.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:05.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:10.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:15.692+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:20.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:25.693+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:30.695+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:35.696+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:40.696+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:45.697+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:50.697+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:11:55.698+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:00.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:05.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:10.700+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:15.699+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:20.700+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:25.701+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:30.702+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:35.703+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:40.703+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:45.704+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:50.704+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:12:55.705+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:00.705+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:05.706+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:10.707+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:15.706+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:20.707+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:25.708+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:30.709+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:35.709+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:40.710+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:45.710+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:50.711+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:13:55.711+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:00.712+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:05.713+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:10.712+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:15.713+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:20.714+0000 7f556b926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:25.714+0000 7f556b926880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:25.714+0000 7f556b926880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:25.714+0000 7f556b926880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2023-09-30T00:47:10.510 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1244, in test_multipart_object_sync 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.511 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-61 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/ 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/ 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/ 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:25 GMT 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/ 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:O7Eb8B+aRqqkqraTx51k4tdHNTg= 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O7Eb8B+aRqqkqraTx51k4tdHNTg=', 'Content-Length': '0'} 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006c7c070353728908-0065175a51-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=dmuhsu-62 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-62/ 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-62/ 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-62/ 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:25 GMT 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:/dmuhsu-62/ 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:80Grp2gCLGTnu92nwIHbVpxTxSE= 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:80Grp2gCLGTnu92nwIHbVpxTxSE=', 'Content-Length': '0'} 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c79aca77685d0a46-0065175a51-14ab-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=dmuhsu-63 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-63/ 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-63/ 2023-09-30T00:47:10.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-63/ 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.516 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:25 GMT 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:/dmuhsu-63/ 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mt6wq04OFnPbU+9JPmubOS7ejdE= 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mt6wq04OFnPbU+9JPmubOS7ejdE=', 'Content-Length': '0'} 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007965a2f919c84bbd-0065175a51-1ea0-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploads 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploads 2023-09-30T00:47:10.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploads 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:POST 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.518 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:25 GMT 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?uploads 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:XY9wXvnG02UHqB2kas3vG1yC/EE= 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:25 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XY9wXvnG02UHqB2kas3vG1yC/EE=', 'Content-Length': '0'} 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000085707f87a3c98815-0065175a51-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dmuhsu-61MULTIPART2~Bta-4Ct08R581PcClvc_t-Hrubf8obn' 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=1 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:27 GMT 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?partNumber=1&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:IRcPN6LH/QV08ePjdOm0lWQAuqw= 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:IRcPN6LH/QV08ePjdOm0lWQAuqw='} 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c2aed21dbc1b2d7-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=2 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-30T00:47:10.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:27 GMT 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?partNumber=2&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hY0NAs+DaLCWRL6DE5uhTnbyH1o= 2023-09-30T00:47:10.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hY0NAs+DaLCWRL6DE5uhTnbyH1o='} 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082c20d3318db3b68-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=3 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:27 GMT 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?partNumber=3&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:PFYoSQCueu3zY5v6eSVVcIfHq38= 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:27 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PFYoSQCueu3zY5v6eSVVcIfHq38='} 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001433115c78a5b09b-0065175a53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn&partNumber=4 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:28 GMT 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?partNumber=4&uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:FVNFRTDnW72fRPCR3IHUqtafLgQ= 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FVNFRTDnW72fRPCR3IHUqtafLgQ='} 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b644bb9ddbfc82a-0065175a54-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests:GET 2023-09-30T00:47:10.527 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:28 GMT 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:p9Fj/tsOo6C1m2HW4WpvEtWtkOE= 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:p9Fj/tsOo6C1m2HW4WpvEtWtkOE=', 'Content-Length': '0'} 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab2caed15863c21-0065175a54-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 1 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.529 INFO:tasks.rgw_multisite_tests: 3 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: 4 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:POST 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:text/xml 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:14:28 GMT 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:/dmuhsu-61/MULTIPART?uploadId=2~Bta-4Ct08R581PcClvc_t-Hrubf8obn 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:B/9qoAhPI3gA5vLtsT44co2iqzk= 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:14:28 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:B/9qoAhPI3gA5vLtsT44co2iqzk=', 'Content-Length': '460'} 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f02c3b158f08dd34-0065175a54-1e9c-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Fri, 29 Sep 2023 23:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi175.front.sepia.ceph.com:8000/dmuhsu-61/MULTIPARTdmuhsu-61MULTIPART 2023-09-30T00:47:10.531 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029265.852881_493.1", 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:14:25.852881Z" 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:14:28.318+0000 7f8fd2013880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:14:28.319+0000 7f8fd2013880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696029265.852881_493.1", 2023-09-30T00:47:10.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:14:28Z" 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.540 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.541 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.542 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029265.852881_493.1", 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.543 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.544 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:14:28Z" 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029265.852881_493.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:25.852881Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:28.426+0000 7f4d53410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:28.427+0000 7f4d53410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029265.852881_493.1", 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:14:28Z" 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.552 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.553 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.554 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029265.852881_493.1", 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:25.852881Z", 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.555 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.556 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:14:28Z" 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029265.852881_493.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:25.852881Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029265.852881_493.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-61 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-61 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-61 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:28.643+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:33.644+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:38.645+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:43.647+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:48.648+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:53.649+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:14:58.651+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:03.652+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:08.653+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:13.653+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:18.654+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:23.656+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:28.658+0000 7fe519c9d880 1 waiting for incremental sync to catch up: 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000005.44.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:33.660+0000 7fe519c9d880 1 bucket sync caught up with source: 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000011.48.6, , , , , ] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:33.660+0000 7fe519c9d880 0 bucket checkpoint complete 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-61 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:33.791+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:38.792+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:43.792+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:48.793+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:53.793+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:15:58.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:03.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:08.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:13.794+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:18.795+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:23.797+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:28.797+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:33.798+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:38.798+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:43.799+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:48.799+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:53.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:16:58.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:03.801+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:08.800+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:13.801+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:18.802+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:23.803+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:28.803+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:33.804+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:38.804+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:43.805+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:48.806+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:53.806+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:17:58.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:03.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:08.807+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:13.808+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:18.808+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:23.809+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:28.810+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:33.811+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:38.811+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:43.812+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:48.812+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:53.813+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:18:58.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:03.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:08.814+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:13.815+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:18.815+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:23.816+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:28.817+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:33.817+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:38.818+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:43.818+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:48.819+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:53.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:19:58.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:03.820+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:08.821+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:13.821+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:18.822+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:23.823+0000 7f797e926880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:28.823+0000 7f797e926880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:28.823+0000 7f797e926880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:28.823+0000 7f797e926880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.568 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1310, in test_bucket_index_log_trim 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1306, in make_test_bucket 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.569 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-65 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/ 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/ 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/ 2023-09-30T00:47:10.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/ 2023-09-30T00:47:10.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Ie+XQiUtDMkkXKcNuzcGV/VZy6g= 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ie+XQiUtDMkkXKcNuzcGV/VZy6g=', 'Content-Length': '0'} 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a1e33b54930e172b-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/ 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/ 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/ 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/ 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9e2a97e18443a76-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/a 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/a 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/a 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.574 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/a 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9wFgG6E1nADkzOTX5XiQHthi+sg= 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9wFgG6E1nADkzOTX5XiQHthi+sg='} 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccebaeffd6b0f8af-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/ 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/ 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/ 2023-09-30T00:47:10.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/ 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.576 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb2f8c66909a6a08-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/b 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/b 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/b 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/b 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:AkP8GMaXXlisDMeYWOkFd5d7aBE= 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:AkP8GMaXXlisDMeYWOkFd5d7aBE='} 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab94224ce6620e71-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/ 2023-09-30T00:47:10.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/ 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/ 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.579 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/ 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009066bd9b50bbc11c-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/c 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/c 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/c 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/c 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.581 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zWFlj7WJ7MRN6LKxQxRI4M1NVzE= 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zWFlj7WJ7MRN6LKxQxRI4M1NVzE='} 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034727b71be43be0c-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/ 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/ 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/ 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/ 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4= 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:tUYnhx83n5Gxws3U8y6ytjMhHN4=', 'Content-Length': '0'} 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f482860d78577557-0065175bbd-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-65/d 2023-09-30T00:47:10.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-65/d 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-65/d 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.584 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:20:29 GMT 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:/dmuhsu-65/d 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Kki+7iRBfqTN9cUc+OKemU8fXws= 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:20:29 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Kki+7iRBfqTN9cUc+OKemU8fXws='} 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9d6f241f61d3d03-0065175bbd-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:20:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029629.500136_513.1", 2023-09-30T00:47:10.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:20:29.500136Z" 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:20:29.821+0000 7f14dc49d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:20:29.822+0000 7f14dc49d880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696029629.500136_513.1", 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.593 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029629.500136_513.1", 2023-09-30T00:47:10.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.598 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.500136_513.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.500136Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:29.929+0000 7f493e737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:29.930+0000 7f493e737880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029629.490601_510.1", 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:20:29.490601Z", 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.605 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029629.490601_510.1", 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:20:29.490601Z", 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.610 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:20:29Z" 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.490601_510.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.490601Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.490601_510.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696029629.500136_513.1 target=1_1696029629.490601_510.1 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:35.052+0000 7f9a37526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:35.053+0000 7f9a37526880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029629.500136_513.1", 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:20:35Z" 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.617 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.618 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029629.500136_513.1", 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:20:29.500136Z", 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.622 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:20:35Z" 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029629.500136_513.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:20:29.500136Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029629.500136_513.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-65 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-65 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-65 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:35.293+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:40.293+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:45.294+0000 7fd426c15880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:50.296+0000 7fd426c15880 1 bucket sync caught up with source: 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, 00000000001.14.6, , , , 00000000001.43.6, , , 00000000001.48.6, , 00000000001.43.6] 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, 00000000001.14.6, , , , 00000000001.43.6, , , 00000000001.48.6, , 00000000001.43.6] 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:50.296+0000 7fd426c15880 0 bucket checkpoint complete 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-65 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:50.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:20:55.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:00.417+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:05.416+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:10.417+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:15.418+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:20.419+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:25.420+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:30.421+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:35.421+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:40.422+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:45.423+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:50.423+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:21:55.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:00.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:05.424+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:10.425+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:15.425+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:20.427+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:25.427+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:30.428+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:35.429+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:40.429+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:45.430+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:50.430+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:22:55.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:00.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:05.431+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:10.432+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:15.432+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:20.434+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:25.434+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:30.435+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:35.436+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:40.436+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:45.437+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:50.437+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:23:55.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:00.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:05.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:10.438+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:15.440+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:20.441+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:25.441+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:30.442+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:35.442+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:40.443+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:45.443+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:50.444+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:24:55.445+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:00.444+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:05.445+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:10.446+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:15.447+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:20.447+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:25.448+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:30.448+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:35.449+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:40.450+0000 7f3710b29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:45.450+0000 7f3710b29880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:45.450+0000 7f3710b29880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:45.450+0000 7f3710b29880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.631 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1364, in test_bucket_reshard_index_log_trim 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1360, in make_test_bucket 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.632 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-66 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/ 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/ 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/ 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/ 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:GIpKm0hdZDG8KcSLCJaj3M7tY4Y= 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GIpKm0hdZDG8KcSLCJaj3M7tY4Y=', 'Content-Length': '0'} 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007c49a2a09b832d45-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/ 2023-09-30T00:47:10.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/ 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/ 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.635 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/ 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007addb06af55ece7-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/a 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/a 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/a 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/a 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:a3CttVooxs+Bqx5an7ZPPTIwc0s= 2023-09-30T00:47:10.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:a3CttVooxs+Bqx5an7ZPPTIwc0s='} 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2d687e73992282c-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/ 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/ 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/ 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/ 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e5f0e39ebc22b91-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/b 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/b 2023-09-30T00:47:10.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/b 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.640 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/b 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mJNYFCj0QBi2XPmMv+S/VNFuOwA= 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mJNYFCj0QBi2XPmMv+S/VNFuOwA='} 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3747643583233cf-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/ 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/ 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/ 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/ 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-30T00:47:10.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ced8b3d82eaee918-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/c 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/c 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/c 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/c 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9abb6l1e9ozB++GIdXBDYQvkeLM= 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9abb6l1e9ozB++GIdXBDYQvkeLM='} 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000122d23955a421e02-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/ 2023-09-30T00:47:10.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/ 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/ 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.645 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/ 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4= 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:L3YZXeOMnWvoGQNWPyQA6AU4Tw4=', 'Content-Length': '0'} 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000627e8a37b95007d0-0065175cf9-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-66/d 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-66/d 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-66/d 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:25:45 GMT 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:/dmuhsu-66/d 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.647 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:xOAGigup0MI/AdPi1llhrsIZ74k= 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:25:45 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:xOAGigup0MI/AdPi1llhrsIZ74k='} 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb2de5e677ed2b7f-0065175cf9-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029945.497737_530.1", 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:25:45.497737Z" 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:25:45.793+0000 7efcdd615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:25:45.794+0000 7efcdd615880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696029945.497737_530.1", 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:25:45Z" 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.656 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029945.497737_530.1", 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:25:45Z" 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.661 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029945.497737_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:25:45.497737Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:45.913+0000 7f197eda2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:45.914+0000 7f197eda2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696029945.497737_530.1", 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:25:45Z" 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029945.497737_530.1", 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:25:45.497737Z", 2023-09-30T00:47:10.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.672 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:25:45Z" 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696029945.497737_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:25:45.497737Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696029945.497737_530.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-66 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-66 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-66 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:46.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:51.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:25:56.156+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:01.157+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:06.157+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:11.158+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:16.160+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:21.160+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:26.161+0000 7f5f7992d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:31.163+0000 7f5f7992d880 1 bucket sync caught up with source: 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, 00000000001.26.6, , , , 00000000001.51.6, , , 00000000001.27.6, , 00000000001.137.6] 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, 00000000001.26.6, , , , 00000000001.51.6, , , 00000000001.27.6, , 00000000001.137.6] 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:31.163+0000 7f5f7992d880 0 bucket checkpoint complete 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-66 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-66', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-66 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:31.298+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:36.298+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:41.299+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:46.299+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:51.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:26:56.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:01.300+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:06.301+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:11.302+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:16.303+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:21.304+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:26.304+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:31.305+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:36.305+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:41.306+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:46.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:51.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:27:56.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:01.307+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:06.308+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:11.309+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:16.310+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:21.311+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:26.311+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:31.312+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:36.312+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:41.313+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:46.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:51.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:28:56.314+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:01.315+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:06.315+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:11.317+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:16.317+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:21.318+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:26.318+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:31.319+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:36.319+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:41.320+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:46.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:51.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:29:56.321+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:01.322+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:06.322+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:11.324+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:16.324+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:21.325+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:26.326+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:31.326+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:36.327+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:41.327+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:46.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:51.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:30:56.328+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:01.329+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:06.330+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:11.331+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:16.331+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:21.332+0000 7f971d511880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:26.332+0000 7f971d511880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:26.332+0000 7f971d511880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:26.333+0000 7f971d511880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.682 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1447, in test_bucket_reshard_incremental 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/ 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/ 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/ 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.684 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:26 GMT 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/ 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:2+cFvOnqzNH5S0Ko//aJGpyHO9Q= 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:26 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2+cFvOnqzNH5S0Ko//aJGpyHO9Q=', 'Content-Length': '0'} 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000094f8adcbd0d40551-0065175e4e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-67 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696030286.374800_534.1", 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:31:26.374800Z" 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:31:26.579+0000 7f3629617880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:31:26.580+0000 7f3629617880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696030286.374800_534.1", 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:31:26Z" 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1696030286.374800_534.1", 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.698 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:31:26Z" 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.374800_534.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.374800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:26.689+0000 7fe93f91e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:26.690+0000 7fe93f91e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696030286.367071_531.1", 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:31:26.367071Z", 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:31:26Z" 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.705 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.707 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1696030286.367071_531.1", 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:31:26.367071Z", 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.710 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:31:26Z" 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.367071_531.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.367071Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.367071_531.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696030286.374800_534.1 target=1_1696030286.367071_531.1 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:31.805+0000 7f8238244880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:31.806+0000 7f8238244880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696030286.374800_534.1", 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:31:31Z" 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1696030286.374800_534.1", 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:31:26.374800Z", 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.723 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:31:31Z" 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030286.374800_534.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:26.374800Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030286.374800_534.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/ 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/ 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/ 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/ 2023-09-30T00:47:10.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c491cbe66d7efc99-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/a 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/a 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/a 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/a 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:8q5FYG2fbz8ivBNN7FYARQ0JwbU= 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8q5FYG2fbz8ivBNN7FYARQ0JwbU='} 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fab3562cbcf3c34c-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/ 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/ 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/ 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.728 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/ 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002162e53fc1e66865-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/b 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/b 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/b 2023-09-30T00:47:10.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/b 2023-09-30T00:47:10.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:DZeewFRffr6GORt3/J/Tf7jhvSI= 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DZeewFRffr6GORt3/J/Tf7jhvSI='} 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e847ba09d918dae-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/ 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/ 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/ 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/ 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5508aa7203e5071-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/c 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/c 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/c 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.733 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/c 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W9auZ3lLrjb877opk6Er/NWMMXY= 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W9auZ3lLrjb877opk6Er/NWMMXY='} 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005829d57de4589173-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/ 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/ 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/ 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.735 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/ 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY= 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Q3PT5VVubn2Iaig17EwK5judnLY=', 'Content-Length': '0'} 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e40a426f2b01b03-0065175e53-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-67/d 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-67/d 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-67/d 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:31:31 GMT 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:/dmuhsu-67/d 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hvuoZ4KrccTPkKJxcDnf1hnrIWk= 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:31 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hvuoZ4KrccTPkKJxcDnf1hnrIWk='} 2023-09-30T00:47:10.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5124fe1d62ed650-0065175e53-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-67 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-67 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-67 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:32.129+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:37.129+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:42.130+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:47.131+0000 7f7032b4b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:52.131+0000 7f7032b4b880 1 bucket sync caught up with source: 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, 00000000001.17.6, , , , 00000000001.48.6, , , 00000000001.31.6, , 00000000001.46.6] 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, 00000000001.17.6, , , , 00000000001.48.6, , , 00000000001.31.6, , 00000000001.46.6] 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:52.131+0000 7f7032b4b880 0 bucket checkpoint complete 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-67 2023-09-30T00:47:10.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-67', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-67 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:52.258+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:31:57.259+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:02.259+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:07.260+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:12.261+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:17.262+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:22.262+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:27.263+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:32.263+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:37.264+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:42.264+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:47.265+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:52.265+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:32:57.266+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:02.266+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:07.268+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:12.268+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:17.269+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:22.270+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:27.270+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:32.271+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:37.271+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:42.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:47.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:52.272+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:33:57.273+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:02.273+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:07.274+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:12.275+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:17.275+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:22.276+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:27.277+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:32.277+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:37.278+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:42.279+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:47.278+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:52.279+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:34:57.280+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:02.280+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:07.281+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:12.282+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:17.283+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:22.283+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:27.284+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:32.284+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:37.285+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:42.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:47.285+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:52.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:35:57.286+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:02.288+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:07.288+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:12.289+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:17.290+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:22.290+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:27.291+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:32.292+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:37.292+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:42.293+0000 7f67a3b23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:47.292+0000 7f67a3b23880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:47.292+0000 7f67a3b23880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:47.292+0000 7f67a3b23880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.745 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1498, in test_bucket_reshard_full 2023-09-30T00:47:10.746 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.748 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:47 GMT 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ikUh7aGPyCAkgcUlsuHq6TPV/PE= 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:47 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ikUh7aGPyCAkgcUlsuHq6TPV/PE=', 'Content-Length': '0'} 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009be27aadf440ebbc-0065175f8f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-68 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696030607.338642_550.1", 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:36:47.338642Z" 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:36:47.557+0000 7f22e8b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:36:47.558+0000 7f22e8b36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696030607.338642_550.1", 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:36:47Z" 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.757 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1696030607.338642_550.1", 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.762 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:36:47Z" 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030607.338642_550.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:36:47.338642Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:47.660+0000 7f499cc30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:36:47.661+0000 7f499cc30880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696030607.338642_550.1", 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:36:47Z" 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.769 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1696030607.338642_550.1", 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:36:47.338642Z", 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:36:47Z" 2023-09-30T00:47:10.774 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696030607.338642_550.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:36:47.338642Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696030607.338642_550.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-29T23:36:47.697+0000 7f6fade67640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 89189) UID: 0 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-29T23:36:47.697+0000 7f6fb4212880 -1 shutting down 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2023-09-30T00:47:10.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi175.stdout: INFO: 2023-09-29T23:36:53.799+0000 7f0f322db640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 90724) UID: 0 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi175.stdout: INFO: 2023-09-29T23:36:53.799+0000 7f0f38434880 -1 shutting down 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~6s 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3310a1464593303-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/a 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/a 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/a 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/a 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:rs7k1hOIkKUKL2oGz64j34ACdAk= 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:rs7k1hOIkKUKL2oGz64j34ACdAk='} 2023-09-30T00:47:10.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6fae89d09930162-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.780 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060ba954bf05675c8-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/b 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/b 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/b 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.782 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/b 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9Mie76D5hf86w9ZhZicSAezy/gs= 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9Mie76D5hf86w9ZhZicSAezy/gs='} 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a321b37be698c8b7-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.784 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d00ec4d9b3039945-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/c 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/c 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/c 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/c 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ml+cigcke75TEI/XWu772YQJx58= 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ml+cigcke75TEI/XWu772YQJx58='} 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c956569061445647-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.787 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg= 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:uRa5g2nuhSvPY4cgQptsTX/WZCg=', 'Content-Length': '0'} 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009263f87a790ee49c-0065175f9b-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/d 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/d 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/d 2023-09-30T00:47:10.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:36:59 GMT 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/d 2023-09-30T00:47:10.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Ebq6iDgGrz0AJ2GQlnMtmJD9HEI= 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:36:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Ebq6iDgGrz0AJ2GQlnMtmJD9HEI='} 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5ba6400891084f3-0065175f9b-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:36:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dmuhsu-68', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dmuhsu-68', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dmuhsu-68 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-68 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 4 2023-09-30T00:47:10.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.791 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd27de35143e6b2e-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/e 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/e 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/e 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.793 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/e 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:HJee3Nwa98e62BEMM4sXn5aKxXo= 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HJee3Nwa98e62BEMM4sXn5aKxXo='} 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f979fa10883c8c4d-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-30T00:47:10.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ff575d95843c665-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/f 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/f 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/f 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/f 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ue0NjbW0XQQebi55AX0EPt3uFcY= 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ue0NjbW0XQQebi55AX0EPt3uFcY='} 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e13d2aa9e886caaa-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.798 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83affcf51f89f2d-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/g 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/g 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/g 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/g 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:O22dzXi3GGeBrWbw6UDzgYceglA= 2023-09-30T00:47:10.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:O22dzXi3GGeBrWbw6UDzgYceglA='} 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a80c2556caa0c810-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/ 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/ 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/ 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.801 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/ 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4= 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFFgxK9Y+9iacRuVlptmrWVTSi4=', 'Content-Length': '0'} 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113e0066b620e476-0065175f9c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-68/h 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-68/h 2023-09-30T00:47:10.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-68/h 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.803 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:37:00 GMT 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:/dmuhsu-68/h 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:UaxC+54cQbV+RqY0hm5G4dRWbUc= 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:37:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UaxC+54cQbV+RqY0hm5G4dRWbUc='} 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007345730c5bf46598-0065175f9c-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:37:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:10.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: anonymous 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-09-30T00:47:10.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > curl http://smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8001: Connection refused 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > curl http://smithi175.front.sepia.ceph.com:8001 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:10.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: anonymous 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-68 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-68 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-68 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:02.624+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:07.625+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:12.626+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:17.626+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:22.627+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:27.627+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:32.628+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:37.628+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:42.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:47.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:52.629+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:37:57.630+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:02.631+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:07.632+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:12.632+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:17.633+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:22.633+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:27.634+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:32.634+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:37.635+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:42.635+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:47.636+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:52.637+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:38:57.637+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:02.638+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:07.639+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:12.640+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:17.641+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:22.642+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:27.642+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:32.643+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:37.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:42.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:47.644+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:52.645+0000 7ff739e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:57.647+0000 7ff739e0f880 1 bucket sync caught up with source: 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [00000000003.27.6, 00000000002.36.6, 00000000002.28.6] 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [00000000003.27.6, 00000000002.36.6, 00000000002.28.6] 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:57.647+0000 7ff739e0f880 0 bucket checkpoint complete 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-68 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:39:57.790+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:02.790+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:07.791+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:12.792+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:17.792+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:22.793+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:27.793+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:32.794+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:37.795+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:42.794+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:47.795+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:52.796+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:40:57.797+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:02.797+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:07.798+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:12.798+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:17.799+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:22.799+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:27.800+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:32.800+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:37.801+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:42.801+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:47.802+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:52.802+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:41:57.803+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:02.804+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:07.804+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:12.805+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:17.806+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:22.806+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:27.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:32.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:37.807+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:42.808+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:47.808+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:52.810+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:42:57.810+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:02.811+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:07.811+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:12.812+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:17.813+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:22.813+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:27.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:32.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:37.814+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:42.815+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:47.815+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:52.816+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:43:57.817+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:02.817+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:07.818+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:12.818+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:17.819+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:22.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:27.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:32.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:37.820+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:42.821+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:47.822+0000 7f544ed33880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:52.823+0000 7f544ed33880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:52.823+0000 7f544ed33880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:52.823+0000 7f544ed33880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2023-09-30T00:47:10.817 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1591, in test_bucket_sync_run_basic_incremental 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.818 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+B6HiDMlAfTJpJt9KANVvpYAmBw= 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+B6HiDMlAfTJpJt9KANVvpYAmBw=', 'Content-Length': '0'} 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000bf36c8c04e95260-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-69 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031093.225022_558.1", 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:44:53.225022Z" 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:44:53.421+0000 7f4f5d067880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:44:53.422+0000 7f4f5d067880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031093.225022_558.1", 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:44:53Z" 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031093.225022_558.1", 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-30T00:47:10.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:44:53Z" 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031093.225022_558.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:44:53.225022Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:53.522+0000 7fb56cb32880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:53.523+0000 7fb56cb32880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031093.225022_558.1", 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:44:53Z" 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.841 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.842 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031093.225022_558.1", 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:44:53.225022Z", 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:44:53Z" 2023-09-30T00:47:10.846 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031093.225022_558.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:44:53.225022Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031093.225022_558.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "objs": [ 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000001", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000005", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000012", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000004", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000011", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000000", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000010", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000003", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000013", 2023-09-30T00:47:10.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000002", 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "obj00000000000000000006" 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b21ce563c5bf3ecb-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000002 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000002 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000002 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.851 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000002 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:d/NSJ1QX1VsJkX7P6SSbux+SHP0= 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d/NSJ1QX1VsJkX7P6SSbux+SHP0='} 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049015d764d39b796-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.853 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac0d6e659b629782-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000001 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000001 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000001 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000001 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:GwxN7OJxohZbXwdAmdv7zAVvZSg= 2023-09-30T00:47:10.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GwxN7OJxohZbXwdAmdv7zAVvZSg='} 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005668f7923108e714-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000610c0a3fb6bb20a4-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000003 2023-09-30T00:47:10.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000003 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000003 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.858 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000003 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3t0Acpx1qOgBbMo5CUk9SvaDPaU= 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3t0Acpx1qOgBbMo5CUk9SvaDPaU='} 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090353d9c6ded4433-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.860 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000619209e4ef4875df-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000004 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000004 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000004 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000004 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:kQRvf9BFur9vbZC6DWyxRT6uA8g= 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kQRvf9BFur9vbZC6DWyxRT6uA8g='} 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ab1984f584774fb-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.863 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090b3bc1e4d3a5c49-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000006 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000006 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000006 2023-09-30T00:47:10.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000006 2023-09-30T00:47:10.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zE4YVeYneOYbWCFs5rkDNPTGpnE= 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zE4YVeYneOYbWCFs5rkDNPTGpnE='} 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a560917238603f5-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000175d9877261392a9-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000011 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000011 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000011 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.868 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000011 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:OIPxxFwM1bBPsI7vYkMHU4d3GGw= 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:OIPxxFwM1bBPsI7vYkMHU4d3GGw='} 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075095bb29ea29963-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.870 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058c69456c750c7ce-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000010 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000010 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000010 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000010 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:aFR71gJbLKxfhTgvJ1iTO1D863E= 2023-09-30T00:47:10.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:aFR71gJbLKxfhTgvJ1iTO1D863E='} 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000152f4ce9d4085105-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cad785acebd236f-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000000 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000000 2023-09-30T00:47:10.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000000 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.875 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000000 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:jmsxus/u/8/c26sUYrcWKGWbvnQ= 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jmsxus/u/8/c26sUYrcWKGWbvnQ='} 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c818581a4fd04b-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000705eb733f3224c89-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000013 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000013 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000013 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000013 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:J1dQvo5Tfgt8Dqwyw0siXIjZWMY= 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:J1dQvo5Tfgt8Dqwyw0siXIjZWMY='} 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c4eac81b10524bc-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/ 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/ 2023-09-30T00:47:10.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/ 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.880 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/ 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34= 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wfj+bibd0ANvGMt7K2+moVEnq34=', 'Content-Length': '0'} 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8ce1f6e05adbe42-0065176175-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-69/obj00000000000000000012 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-69/obj00000000000000000012 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-69/obj00000000000000000012 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:44:53 GMT 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:/dmuhsu-69/obj00000000000000000012 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:LdHjAi1Aa22FyA4XUIjIvySDDs4= 2023-09-30T00:47:10.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:44:53 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LdHjAi1Aa22FyA4XUIjIvySDDs4='} 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006004e5c1c2fe192a-0065176175-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-69 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-69 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-69 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:54.139+0000 7f33d5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:44:59.140+0000 7f33d5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:04.142+0000 7f33d5729880 1 bucket sync caught up with source: 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [00000000001.54.6, , 00000000001.32.6, 00000000001.31.6, 00000000001.43.6, 00000000001.48.6, 00000000001.47.6, 00000000001.51.6, 00000000001.28.6, 00000000001.30.6, 00000000001.52.6] 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [00000000001.54.6, , 00000000001.32.6, 00000000001.31.6, 00000000001.43.6, 00000000001.48.6, 00000000001.47.6, 00000000001.51.6, 00000000001.28.6, 00000000001.30.6, 00000000001.52.6] 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:04.142+0000 7f33d5729880 0 bucket checkpoint complete 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-69 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:04.270+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:09.270+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:14.271+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:19.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:24.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:29.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:34.272+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:39.273+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:44.274+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:49.275+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:54.276+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:45:59.276+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:04.277+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:09.277+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:14.279+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:19.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:24.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:29.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:34.280+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:39.281+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:44.282+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:49.283+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:54.283+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:46:59.284+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:04.284+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:09.285+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:14.286+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:19.286+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:24.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:29.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:34.287+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:39.288+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:44.289+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:49.290+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:54.291+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:47:59.291+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:04.292+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:09.292+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:14.293+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:19.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:24.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:29.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:34.294+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:39.295+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:44.296+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:49.297+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:54.297+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:48:59.298+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:04.299+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:09.299+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:14.300+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:19.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:24.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:29.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:34.301+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:39.303+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:44.304+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:49.304+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:54.305+0000 7fc112612880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:59.305+0000 7fc112612880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:59.305+0000 7fc112612880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:59.305+0000 7fc112612880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.890 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1658, in test_zap_init_bucket_sync_run 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.891 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:UMj4/X0lYaJ59XdpOKK7iRzp9ac= 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UMj4/X0lYaJ59XdpOKK7iRzp9ac=', 'Content-Length': '0'} 2023-09-30T00:47:10.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e4446aff7e8b68a2-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-70 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:09:41.280108Z" 2023-09-30T00:47:10.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "0.000000" 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031399.349060_574.1", 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:49:59.349060Z" 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:49:59.556+0000 7f93a6711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:49:59.557+0000 7f93a6711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "", 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031399.349060_574.1", 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:49:59Z" 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031399.349060_574.1", 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.905 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:49:59Z" 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031399.349060_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:49:59.349060Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:59.679+0000 7fcea180f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:49:59.680+0000 7fcea180f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "", 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "0.000000", 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031399.349060_574.1", 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:49:59Z" 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.914 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025381.280108_31.1", 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:09:41.280108Z", 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.915 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031399.349060_574.1", 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:49:59.349060Z", 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:49:59Z" 2023-09-30T00:47:10.920 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696025381.280108_31.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:09:41.280108Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031399.349060_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:49:59.349060Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696025381.280108_31.1', 1: '', 2: '1_1696031399.349060_574.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.922 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020cfaf7a42d4cc37-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/obj11 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/obj11 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/obj11 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/obj11 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qox4yuk+gZKccobO3tqCvYwjmf0= 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qox4yuk+gZKccobO3tqCvYwjmf0='} 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbf8beea3f9c4f59-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.925 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbb293bf633d25b2-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/obj22 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/obj22 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/obj22 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.927 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/obj22 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:DVUNWzUT2oM6Z6N7m0EpBcQzg6M= 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DVUNWzUT2oM6Z6N7m0EpBcQzg6M='} 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021fd4751eeb5f837-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e2c01ab3d94a544-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/obj33 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/obj33 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/obj33 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.930 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/obj33 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:lnb8t+vBhgdbKeIJNV2CNpcZ38k= 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lnb8t+vBhgdbKeIJNV2CNpcZ38k='} 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080d40c7c3fb15f87-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b49374d7276c918f-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/obj44 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/obj44 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/obj44 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/obj44 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hYgpFD6UYowD19KsUosHXZRuiFA= 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hYgpFD6UYowD19KsUosHXZRuiFA='} 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005804958fc9dd4c81-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/ 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/ 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/ 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.935 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/ 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM= 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zNmQhpDmWv+S1NiuvvsNvlR5luM=', 'Content-Length': '0'} 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295da537e09e17ea-00651762a7-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-70/obj55 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-70/obj55 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-70/obj55 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:49:59 GMT 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:/dmuhsu-70/obj55 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.937 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:lqyyIpJKCuIY/PkxBOe4yG4EZWE= 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:49:59 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:lqyyIpJKCuIY/PkxBOe4yG4EZWE='} 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b8915339b7a1aed-00651762a7-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-70 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-70 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-70 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:00.067+0000 7f888649d880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:05.069+0000 7f888649d880 1 bucket sync caught up with source: 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , , , , 00000000001.35.6, 00000000002.23.6, 00000000002.58.6] 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , , , , 00000000001.35.6, 00000000002.23.6, 00000000002.58.6] 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:05.069+0000 7f888649d880 0 bucket checkpoint complete 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-70 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:05.204+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:10.204+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:15.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:20.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:25.205+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:30.206+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:35.206+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:40.207+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:45.208+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:50.209+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:50:55.209+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:00.210+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:05.210+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:10.211+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:15.212+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:20.213+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:25.212+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:30.213+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:35.214+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:40.215+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:45.216+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:50.216+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:51:55.217+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:00.217+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:05.218+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:10.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:15.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:20.219+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:25.220+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:30.220+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:35.221+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:40.222+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:45.223+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:50.223+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:52:55.224+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:00.224+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:05.225+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:10.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:15.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:20.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:25.226+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:30.227+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:35.229+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:40.229+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:45.230+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:50.230+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:53:55.231+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:00.231+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:05.232+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:10.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:15.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:20.233+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:25.234+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:30.234+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:35.235+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:40.236+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:45.237+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:50.237+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:54:55.238+0000 7f5634e0f880 1 waiting to reach incremental sync.. 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:00.238+0000 7f5634e0f880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:00.238+0000 7f5634e0f880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:00.238+0000 7f5634e0f880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:10.945 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1736, in test_bucket_full_sync_after_data_sync_init 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:10.946 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:00 GMT 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.948 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:FPpaQWWHx9DGoJL8l4pa0soUFX8= 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:00 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FPpaQWWHx9DGoJL8l4pa0soUFX8=', 'Content-Length': '0'} 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006ceb52896743f036-00651763d4-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-71 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.760700_581.1", 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.760700Z" 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:55:00.954+0000 7f624694b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:55:00.955+0000 7f624694b880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:10.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:10.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.760700_581.1", 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:10.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:10.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-29T23:55:00Z" 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.957 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.760700_581.1", 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:55:00Z" 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.962 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.760700_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.760700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:01.055+0000 7f3dce11a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:01.056+0000 7f3dce11a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.322172_577.1", 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.322172Z", 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:55:01Z" 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.322172_577.1", 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.322172Z", 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.972 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.973 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:55:01Z" 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.322172_577.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.322172Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.322172_577.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696031700.760700_581.1 target=1_1696031700.322172_577.1 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:06.172+0000 7f468fe0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:06.173+0000 7f468fe0a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.760700_581.1", 2023-09-30T00:47:10.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:10.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-29T23:55:06Z" 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:10.981 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.982 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.983 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.760700_581.1", 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.760700Z", 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.984 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:10.985 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:55:06Z" 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696031700.760700_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.760700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696031700.760700_581.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:10.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-29T23:55:06.199+0000 7f2ca8046640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 115458) UID: 0 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-29T23:55:06.199+0000 7f2cab897880 -1 shutting down 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-30T00:47:10.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054b1adb8e7433ff2-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/obj11 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/obj11 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/obj11 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/obj11 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+Klo3VSZI3aQWRFejn9gl4DSX2k= 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+Klo3VSZI3aQWRFejn9gl4DSX2k='} 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a29e46b0001319a-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:10.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.991 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e052734d08d9737d-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/obj22 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/obj22 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/obj22 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/obj22 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:SSBojpGxdwTui+nE6ru443Qq6os= 2023-09-30T00:47:10.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SSBojpGxdwTui+nE6ru443Qq6os='} 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003efde1059f89e1a5-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceb80da470328394-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/obj33 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/obj33 2023-09-30T00:47:10.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/obj33 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:10.996 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/obj33 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:UIORxCWZ4sBlypUF9TYMld4Uung= 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:UIORxCWZ4sBlypUF9TYMld4Uung='} 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e5269364fa99d98-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:10.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-30T00:47:10.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c35e6e63cbd10b7-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/obj44 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/obj44 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/obj44 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:10.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/obj44 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NY7IzWnfmgCqkdP2Vxce3/BbbDs= 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NY7IzWnfmgCqkdP2Vxce3/BbbDs='} 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a530a6d5a57cb46-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/ 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/ 2023-09-30T00:47:11.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/ 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.001 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/ 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0= 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WHjcPqgU+Gf58WRazE2PSi+eFn0=', 'Content-Length': '0'} 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052d885013ea79282-00651763da-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-71/obj55 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-71/obj55 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-71/obj55 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:55:06 GMT 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:/dmuhsu-71/obj55 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:5KZgMrR/HyPddCgIG8DpEoEXUZI= 2023-09-30T00:47:11.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:55:06 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5KZgMrR/HyPddCgIG8DpEoEXUZI='} 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f9912e5f6ec9123-00651763da-1e9c-test-zone1'), ('Date', 'Fri, 29 Sep 2023 23:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:55:06.510+0000 7f8829811880 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-29T23:55:06.511+0000 7f8829811880 5 RGW-SYNC:data: acquired data sync status lease 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2023-09-30T00:47:11.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: anonymous 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-71 2023-09-30T00:47:11.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-71 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-71 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:07.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:12.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:17.832+0000 7fbaadaba880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:22.834+0000 7fbaadaba880 1 bucket sync caught up with source: 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , , , , 00000000001.36.6, 00000000002.49.6, 00000000002.52.6] 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , , , , 00000000001.36.6, 00000000002.49.6, 00000000002.52.6] 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:22.834+0000 7fbaadaba880 0 bucket checkpoint complete 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-71 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:22.971+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:27.972+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:32.973+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:37.974+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:42.974+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:47.975+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:52.976+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:55:57.976+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:02.977+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:07.977+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:12.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:17.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:22.978+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:27.979+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:32.980+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:37.981+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:42.981+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:47.982+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:52.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:56:57.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:02.983+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:07.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:12.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:17.984+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:22.985+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:27.986+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:32.987+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:37.987+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:42.988+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:47.989+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:52.989+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:57:57.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:02.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:07.991+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:12.990+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:17.991+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:22.992+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:27.993+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:32.994+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:37.994+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:42.995+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:47.995+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:52.996+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:58:57.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:02.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:07.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:12.997+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:17.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:22.998+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:28.000+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:33.000+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:38.001+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:43.001+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:48.002+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:53.003+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-29T23:59:58.003+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:03.004+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:08.005+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:13.004+0000 7f8f16f20880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:18.005+0000 7f8f16f20880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:18.005+0000 7f8f16f20880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:18.005+0000 7f8f16f20880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2023-09-30T00:47:11.013 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1776, in test_resharded_bucket_full_sync_after_data_sync_init 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:11.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:18 GMT 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.016 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:BnHpl12/l9P6GhXTtUoV/BQ16cA= 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BnHpl12/l9P6GhXTtUoV/BQ16cA=', 'Content-Length': '0'} 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000924869d3d71874df-0065176512-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-72 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:00:18.060994Z" 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:18.272+0000 7f7004136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:18.273+0000 7f7004136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:00:18Z" 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:47:11.028 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:00:18Z" 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.052336_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.052336Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.052336_610.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696032018.060994_613.1 target=1_1696032018.052336_610.1 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:23.382+0000 7f8efb215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:23.383+0000 7f8efb215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:00:23Z" 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.037 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.038 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032018.052336_610.1", 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:00:18.052336Z", 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.041 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:00:23Z" 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.052336_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.052336Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.052336_610.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696032018.060994_613.1 target=1_1696032018.052336_610.1 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:28.495+0000 7f1f53044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:28.496+0000 7f1f53044880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.049 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.052 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.053 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.060994_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.060994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:28.618+0000 7f36e3813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:28.619+0000 7f36e3813880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.061 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032018.060994_613.1", 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:00:18.060994Z", 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.064 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:00:28Z" 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032018.060994_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:00:18.060994Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032018.060994_613.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-30T00:00:28.647+0000 7f15e156d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 117059) UID: 0 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi184.stdout: INFO: 2023-09-30T00:00:28.648+0000 7f15e7a10880 -1 shutting down 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.068 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bd9995ac660f80e-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj11 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj11 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj11 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj11 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.070 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:TjFnUhGXV7zXfL6bWtfZKKR5EP8= 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TjFnUhGXV7zXfL6bWtfZKKR5EP8='} 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55af2a2d6760145-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faa72298fb081e77-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj22 2023-09-30T00:47:11.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj22 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj22 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.073 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj22 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:07XVZ4zeGM5viLltumPJa+QcuiY= 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:07XVZ4zeGM5viLltumPJa+QcuiY='} 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000189c68d8a6ddb49f-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000404f9bbdd3b645d6-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj33 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj33 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj33 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj33 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:cKX+wY6FfL9OjKNfuguBqnRLumY= 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cKX+wY6FfL9OjKNfuguBqnRLumY='} 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b604a47afd9b523-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.078 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a4c48568339383-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj44 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj44 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj44 2023-09-30T00:47:11.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj44 2023-09-30T00:47:11.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NVETEW08T6IK+aiYuxQSXj9HBSE= 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NVETEW08T6IK+aiYuxQSXj9HBSE='} 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084a9585a012beba2-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU= 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:W7+C+jP/F+uJhFAiVYjSV60tLiU=', 'Content-Length': '0'} 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000740466b349d549a2-0065176522-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj55 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj55 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj55 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.083 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:34 GMT 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj55 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zqi1NPMGyJJLiilR6b4sE+SryCs= 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:34 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zqi1NPMGyJJLiilR6b4sE+SryCs='} 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eecc0336eca1d9fa-0065176522-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 5 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b03cd72313afdb14-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj17 2023-09-30T00:47:11.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj17 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj17 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.087 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj17 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3hOr8Ni86kZjUHLjCf4MOqLBpdM= 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3hOr8Ni86kZjUHLjCf4MOqLBpdM='} 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a66aef88d3809a6-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.089 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003acd69cc1c84fa3a-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj34 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj34 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj34 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj34 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Sq1Pnvn8EQDWWR5RlIa122ogpzU= 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Sq1Pnvn8EQDWWR5RlIa122ogpzU='} 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064d966516634b619-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.092 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e978061ce7d65cff-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj51 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj51 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj51 2023-09-30T00:47:11.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj51 2023-09-30T00:47:11.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ScG0x3l7NuNw+3/9+mSosMdeh6Y= 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ScG0x3l7NuNw+3/9+mSosMdeh6Y='} 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000901e2d7e1a02ffca-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ffc97d0a0b44849-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj68 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj68 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj68 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.097 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj68 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:kNsTO/xZA+K7Lw5hRET7tUqkVDM= 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kNsTO/xZA+K7Lw5hRET7tUqkVDM='} 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b395d82f96fd0e9-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.099 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c824cd78afe3456-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj85 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj85 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj85 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj85 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:llBx9cQsEDoIz9qn3Pmo3pQ097g= 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:llBx9cQsEDoIz9qn3Pmo3pQ097g='} 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095fa153ad329b96f-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 10 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056d0c6436dcf39a8-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj19 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj19 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj19 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj19 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:PPZQAotqaJiKhMVDkJWEW4Ex1BA= 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:PPZQAotqaJiKhMVDkJWEW4Ex1BA='} 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5f0ff27ca3eb5e9-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.106 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7bb2dad647ecce5-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj38 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj38 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj38 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.108 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj38 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:6FGI6IMyRt2xJi4EUlx5xuGveQE= 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6FGI6IMyRt2xJi4EUlx5xuGveQE='} 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af698ff4393039a5-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057c569aee4267a36-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj57 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj57 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj57 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.111 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj57 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:LN6kc/YCUcuup3D7i2F2UYvS4II= 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:LN6kc/YCUcuup3D7i2F2UYvS4II='} 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8872d03e31fcfce-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.113 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bd32db31b6e6e94-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj76 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj76 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj76 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj76 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Z7U3mF1vLWWYQBbd65eKZ9mCZoQ= 2023-09-30T00:47:11.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Z7U3mF1vLWWYQBbd65eKZ9mCZoQ='} 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003baacc54a5ef5012-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098af0eaab71e2696-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj95 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj95 2023-09-30T00:47:11.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj95 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.118 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj95 2023-09-30T00:47:11.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.119 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NZAiQ+szwSxWjZr9rk54TQnIHpk= 2023-09-30T00:47:11.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NZAiQ+szwSxWjZr9rk54TQnIHpk='} 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042b757c39ae9b40a-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 15 2023-09-30T00:47:11.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.122 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f84019c38fba4dc9-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj23 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj23 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj23 2023-09-30T00:47:11.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj23 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.124 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:JgzDtm1670SWMsERD48qJyc7PiA= 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JgzDtm1670SWMsERD48qJyc7PiA='} 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000273c9ad95ac27944-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d97fcb2e43ecf687-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj46 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj46 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj46 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.127 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj46 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:E265vdwud4+zqObD5Z8RLz5/fE8= 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E265vdwud4+zqObD5Z8RLz5/fE8='} 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7b840a5445ade83-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfec5906ebe32313-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj69 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj69 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj69 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj69 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:A5GW9/C4v654Ny1elANfjTpARXo= 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:A5GW9/C4v654Ny1elANfjTpARXo='} 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d1aacdad02aca47-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.132 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030817e7dc929bf79-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj92 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj92 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj92 2023-09-30T00:47:11.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj92 2023-09-30T00:47:11.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:F4+gW26YtGpH4sj9vH49QMQzKn8= 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F4+gW26YtGpH4sj9vH49QMQzKn8='} 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073ffd653f2a2af82-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014d0272b0836aa6d-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj115 2023-09-30T00:47:11.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj115 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj115 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.137 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj115 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:RtXkO20fJo45PJ6LU8ID9CSpKZY= 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:RtXkO20fJo45PJ6LU8ID9CSpKZY='} 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d3ca37efb32a5f1-0065176523-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 20 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80= 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:x3uIwsL2ttkWSwpSdRmvS0Kqr80=', 'Content-Length': '0'} 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab633dbe534b118e-0065176523-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj29 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj29 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj29 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.141 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:35 GMT 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj29 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:R/oYbwcqbX5nYX8xoIPOSTbN1p4= 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:35 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:R/oYbwcqbX5nYX8xoIPOSTbN1p4='} 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004984ce35b5ea4df8-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000409f28769fb79ea2-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj58 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj58 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj58 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj58 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qRdrm16gIFL0dkpvW8vrJkmEuto= 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qRdrm16gIFL0dkpvW8vrJkmEuto='} 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd3693c95f6a4824-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.146 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002120fe4975a98160-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj87 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj87 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj87 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj87 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.148 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:oc2vABim/HkvkJ7tH1yboPJrtXI= 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oc2vABim/HkvkJ7tH1yboPJrtXI='} 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000126cc538fc5d8206-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bbf225e73d6fafc-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj116 2023-09-30T00:47:11.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj116 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj116 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.151 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj116 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:VIX68o4ioYpi8FqaSbxTgi+ojcQ= 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VIX68o4ioYpi8FqaSbxTgi+ojcQ='} 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091843b523a96be27-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aad03c69ef1f77a-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj145 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj145 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj145 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj145 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zkcfriRZP8ID2z3dO7VKKjeKJ9c= 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zkcfriRZP8ID2z3dO7VKKjeKJ9c='} 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e38c401c8b60f3a-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 25 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6d63d949b7426de-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj31 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj31 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj31 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.158 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj31 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3GIN1CM7rzkMvtrhnzvHxa+tvHk= 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3GIN1CM7rzkMvtrhnzvHxa+tvHk='} 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000062d360b6bb9c9cf-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c64b3f40d321a607-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj62 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj62 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj62 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj62 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9/m3ArO3OpI83AME6zkpjF+6+Ls= 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9/m3ArO3OpI83AME6zkpjF+6+Ls='} 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000527105ad2a94862a-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.163 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009578fbc01dce6a5c-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj93 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj93 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj93 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj93 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:YB7VDPdWCQSdH/FZyTZJX9tc+XY= 2023-09-30T00:47:11.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:YB7VDPdWCQSdH/FZyTZJX9tc+XY='} 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a30f7f7210cb727-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000992c76c587a3ce3f-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj124 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj124 2023-09-30T00:47:11.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj124 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.168 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj124 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hWnRPRo5YGevb8qvYRD7ns70aXo= 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hWnRPRo5YGevb8qvYRD7ns70aXo='} 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f10b6af485267cf-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c6a50d83e1a974a-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj155 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj155 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj155 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj155 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:MrUFwIKF2WSMLH+XWN/TPG7TuXE= 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:MrUFwIKF2WSMLH+XWN/TPG7TuXE='} 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dbd9466fbbff7b4-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-72', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-72 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-72 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 30 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d31e6df899155c5f-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj37 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj37 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj37 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj37 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1n17JKPLoS+fnZpPm8On+gbtR24= 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1n17JKPLoS+fnZpPm8On+gbtR24='} 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f98ee9fe4510b12f-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.177 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000346ad7c0d34ad9da-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj74 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj74 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj74 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj74 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+f9PZDn9QJhOnmR8rucqcDrRbUU= 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+f9PZDn9QJhOnmR8rucqcDrRbUU='} 2023-09-30T00:47:11.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c648c1ae8c27a3d1-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.180 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f6393470731d38-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj111 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj111 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj111 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.182 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj111 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:oNSY0MudyCdgJA8M4eDkbkHDrss= 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oNSY0MudyCdgJA8M4eDkbkHDrss='} 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e949118bd33d85f-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebbe027de036f65e-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj148 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj148 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj148 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj148 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:5Xw2uSlZNdfX6V/JBrE4VsGzsxY= 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5Xw2uSlZNdfX6V/JBrE4VsGzsxY='} 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1c8787e799f5a35-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/ 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/ 2023-09-30T00:47:11.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/ 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.187 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/ 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic= 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:86LPYS43Z3Ob2ZcRr4Lc+OI/2Ic=', 'Content-Length': '0'} 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f678851ceb02209-0065176524-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-72/obj185 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-72/obj185 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-72/obj185 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:00:36 GMT 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:/dmuhsu-72/obj185 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.189 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9tmi15GudFOSUpnk7RcDwoUTrHI= 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:00:36 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9tmi15GudFOSUpnk7RcDwoUTrHI='} 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097134474879bcdae-0065176524-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:36.985+0000 7fc272815880 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:00:36.987+0000 7fc272815880 5 RGW-SYNC:data: acquired data sync status lease 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-30T00:47:11.190 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > curl http://smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: anonymous 2023-09-30T00:47:11.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-72 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-72 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-72 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:38.330+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:43.331+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:48.332+0000 7fa722c19880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:53.334+0000 7fa722c19880 1 bucket sync caught up with source: 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , , 00000000002.62.6, , , , , , 00000000002.85.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.42.6, , , , 00000000001.49.6] 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , , 00000000002.62.6, , , , , , 00000000002.85.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.42.6, , , , 00000000001.49.6] 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:53.334+0000 7fa722c19880 0 bucket checkpoint complete 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-72 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:53.444+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:00:58.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:03.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:08.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:13.445+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:18.446+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:23.447+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:28.448+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:33.449+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:38.449+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:43.450+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:48.451+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:53.451+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:01:58.452+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:03.453+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:08.452+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:13.453+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:18.454+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:23.455+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:28.456+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:33.456+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:38.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:43.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:48.457+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:53.458+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:02:58.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:03.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:08.459+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:13.460+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:18.461+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:23.462+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:28.463+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:33.463+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:38.464+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:43.464+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:48.465+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:53.466+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:03:58.466+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:03.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:08.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:13.467+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:18.468+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:23.469+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:28.470+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:33.470+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:38.471+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:43.471+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:48.472+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:53.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:04:58.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:03.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:08.473+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:13.474+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:18.475+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:23.476+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:28.477+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:33.477+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:38.478+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:43.478+0000 7f05c5729880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:48.479+0000 7f05c5729880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:48.479+0000 7f05c5729880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:48.479+0000 7f05c5729880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2023-09-30T00:47:11.199 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1794, in test_bucket_incremental_sync_after_data_sync_init 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:11.200 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-73/ 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-73/ 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-73/ 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:05:48 GMT 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:/dmuhsu-73/ 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:jqTAnHiOLVIak3kbgGCP9jYORFI= 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jqTAnHiOLVIak3kbgGCP9jYORFI=', 'Content-Length': '0'} 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000feb0cf4c1933e8ff-006517665c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-73 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:05:48.533216Z" 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:05:48.754+0000 7f837c80f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:05:48.755+0000 7f837c80f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.212 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.213 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.214 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.215 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.216 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032348.533216_654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:05:48.533216Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:48.863+0000 7fb1a2612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:48.864+0000 7fb1a2612880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:47:11.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.224 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.225 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.226 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032348.533216_654.1", 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:05:48.533216Z", 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.227 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.228 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:05:48Z" 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032348.533216_654.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:05:48.533216Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032348.533216_654.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-73/ 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-73/ 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-73/ 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:05:48 GMT 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:/dmuhsu-73/ 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Igh6JQfPZ/pQlpTfEGUDStbNbMM= 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Igh6JQfPZ/pQlpTfEGUDStbNbMM=', 'Content-Length': '0'} 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e22b7a8fa05f901-006517665c-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-73/dummy 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-73/dummy 2023-09-30T00:47:11.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-73/dummy 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.232 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:05:48 GMT 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:/dmuhsu-73/dummy 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:TRM6zsrbZzfMREXcssQ8RdrdRqM= 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:05:48 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TRM6zsrbZzfMREXcssQ8RdrdRqM='} 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000965184032a20593e-006517665c-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:05:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-73 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-73 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-73 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:49.092+0000 7f74c67a2880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:54.093+0000 7f74c67a2880 1 bucket sync caught up with source: 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , 00000000001.65.6, , , , , ] 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , 00000000001.65.6, , , , , ] 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:54.093+0000 7f74c67a2880 0 bucket checkpoint complete 2023-09-30T00:47:11.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-73 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:54.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:05:59.224+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:04.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:09.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:14.223+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:19.225+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:24.225+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:29.226+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:34.226+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:39.227+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:44.227+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:49.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:54.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:06:59.228+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:04.229+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:09.229+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:14.231+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:19.232+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:24.232+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:29.233+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:34.233+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:39.234+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:44.234+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:49.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:54.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:07:59.235+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:04.236+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:09.236+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:14.238+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:19.238+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:24.239+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:29.239+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:34.240+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:39.240+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:44.241+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:49.241+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:54.242+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:08:59.242+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:04.243+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:09.243+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:14.245+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:19.245+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:24.246+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:29.247+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:34.247+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:39.248+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:44.248+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:49.249+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:54.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:09:59.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:04.250+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:09.251+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:14.252+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:19.252+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:24.253+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:29.254+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:34.254+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:39.255+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:44.256+0000 7f0500d29880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:49.256+0000 7f0500d29880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:49.256+0000 7f0500d29880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:49.256+0000 7f0500d29880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:11.240 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.241 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:49 GMT 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:iGA8db0jrM4WVV+w5uzviKZkj9M= 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:49 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:iGA8db0jrM4WVV+w5uzviKZkj9M=', 'Content-Length': '0'} 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000cc8f0566d11ea716-0065176789-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-74 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:10:49.319298Z" 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:10:49.522+0000 7f03fd210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:10:49.523+0000 7f03fd210880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032649.310028_655.1", 2023-09-30T00:47:11.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:10:49.310028Z", 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:10:49Z" 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.253 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.254 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.255 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032649.310028_655.1", 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:10:49.310028Z", 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.256 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.257 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:10:49Z" 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.310028_655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.310028Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.310028_655.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696032649.319298_658.1 target=1_1696032649.310028_655.1 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:10:54.632+0000 7fc1f08c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:10:54.633+0000 7fc1f08c0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.265 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.266 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.267 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.268 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.269 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.319298_658.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.319298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:54.763+0000 7f1536329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:54.764+0000 7f1536329880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032649.319298_658.1", 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:10:49.319298Z", 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:10:54Z" 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.282 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032649.319298_658.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:10:49.319298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032649.319298_658.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:47:11.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037e46e668c8719c1-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj11 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj11 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj11 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj11 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:62qD4aFLwtORPmed+ihB2QWdTWA= 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:62qD4aFLwtORPmed+ihB2QWdTWA='} 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003be97d62cca8a85e-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.287 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6dade10bae95a9b-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj22 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj22 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj22 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj22 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.289 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:cIEnzzXxifeKXcZA4bAw4gMymV8= 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cIEnzzXxifeKXcZA4bAw4gMymV8='} 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003655181d3ff1910-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f7faca0703e69da-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj33 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj33 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj33 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.292 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj33 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:y3BnmBBKMwxLtPAnZBF1UCk+Mro= 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:y3BnmBBKMwxLtPAnZBF1UCk+Mro='} 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000140cc8d256761a90-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000698b51ebcd32e513-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj44 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj44 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj44 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.296 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj44 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:dNhop5LZYrXV56j4QRZLh16CMx8= 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:dNhop5LZYrXV56j4QRZLh16CMx8='} 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023b67c9634816d49-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.298 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI= 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Wa8bWEaAAWj+bqSTW374l4QYSlI=', 'Content-Length': '0'} 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046828f503114ac0b-006517678e-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj55 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj55 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj55 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:54 GMT 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj55 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:XbDaf2hZkBhOT2yqE0dAHuOJvlY= 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:54 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XbDaf2hZkBhOT2yqE0dAHuOJvlY='} 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3ea81b5e22c1449-006517678e-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 5 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.302 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a3659c0c3a4c5dc-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj17 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj17 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj17 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj17 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Mdu8AGvY12qsW2JL8l4He1AbloA= 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mdu8AGvY12qsW2JL8l4He1AbloA='} 2023-09-30T00:47:11.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d5f77f0582df06-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.305 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071f2ccaa2eebe443-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj34 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj34 2023-09-30T00:47:11.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj34 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.307 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj34 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:CmLex/Glrcl1b7Y4jJLIkdKAPog= 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CmLex/Glrcl1b7Y4jJLIkdKAPog='} 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aca50992db9dbbc5-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007be09e56bff61175-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj51 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj51 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj51 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj51 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:eNwoszaQPPWWyvSdMuycrFwrSjo= 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:eNwoszaQPPWWyvSdMuycrFwrSjo='} 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b3ea9de510a015a-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.312 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075dd1ea23899c39d-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj68 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj68 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj68 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj68 2023-09-30T00:47:11.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:flM1GxX6q+2G1K+XvFXOG2Ony2U= 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:flM1GxX6q+2G1K+XvFXOG2Ony2U='} 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb525a674a14ec58-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ad021644141784c-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj85 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj85 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj85 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.317 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj85 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:cyOr8rsuPYwoNQzrB7QuZd8Q9i4= 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:cyOr8rsuPYwoNQzrB7QuZd8Q9i4='} 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e1cfde0ca4d4fa4-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 10 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bb0bd26d1f29b1-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj19 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj19 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj19 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj19 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:F1uvkGotPjKlAPEpcqvnPrSlgKk= 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:F1uvkGotPjKlAPEpcqvnPrSlgKk='} 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022de351bc9691382-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.323 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030b952396aa5e665-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj38 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj38 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj38 2023-09-30T00:47:11.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj38 2023-09-30T00:47:11.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9jjAv494blBc1ca2ZBawjJcnaOE= 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9jjAv494blBc1ca2ZBawjJcnaOE='} 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958efba8e61e9ecd-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053b5356becb911ad-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj57 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj57 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj57 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.328 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj57 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:DOkja9SSR3c2SavQ0FgNP52y8yk= 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:DOkja9SSR3c2SavQ0FgNP52y8yk='} 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab6b6f2206ef42d9-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.330 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6d70504e1bbcb2e-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj76 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj76 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj76 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj76 2023-09-30T00:47:11.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:E/qCIRFlWAwqUZcbrkjyp3X2UAE= 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E/qCIRFlWAwqUZcbrkjyp3X2UAE='} 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1c477f54c2fbef0-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dff7f81ea1bc897-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj95 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj95 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj95 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.335 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj95 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:epFhBGwOK2lCOJigW0ReY43XnLI= 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:epFhBGwOK2lCOJigW0ReY43XnLI='} 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005026b041f76211e3-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 15 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5730d571318cb94-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj23 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj23 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj23 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj23 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:r8bFKskplvTyfWZLIHxwzD9bniA= 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:r8bFKskplvTyfWZLIHxwzD9bniA='} 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6f22926ad11510f-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.341 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584af0e15eee66d8-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj46 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj46 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj46 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj46 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.343 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:EenB18XUbXrQ+j0qqzaDjTdh3JY= 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EenB18XUbXrQ+j0qqzaDjTdh3JY='} 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088ba695be9cee062-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2408a70ed83503a-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj69 2023-09-30T00:47:11.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj69 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj69 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.346 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj69 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:u/900EvxScsSO8u5/sG3u6wQ8mk= 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:u/900EvxScsSO8u5/sG3u6wQ8mk='} 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033b5ef9f0c36594c-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000402f2019ab0d9c4d-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj92 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj92 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj92 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj92 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:X7h+9wl6rpQ1KuU8l68lx0nKgvA= 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:X7h+9wl6rpQ1KuU8l68lx0nKgvA='} 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004f600865160ff78-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.351 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI= 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:9U9BZQDUDjcmjgOKbSh3ywvcsgI=', 'Content-Length': '0'} 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fe0ca247e6be0de-006517678f-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj115 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj115 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj115 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:55 GMT 2023-09-30T00:47:11.353 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj115 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:GR6M4QtDGmKPnE7ibU655NoeuA0= 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:55 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:GR6M4QtDGmKPnE7ibU655NoeuA0='} 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a719c8bc7f447d6-006517678f-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 20 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7a4f54d7f385a76-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj29 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj29 2023-09-30T00:47:11.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj29 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.357 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj29 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:JWwEoavustB0FYr3HSY8evBIgPM= 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:JWwEoavustB0FYr3HSY8evBIgPM='} 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079f15e79431f40f2-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.359 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000427ce68922140965-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj58 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj58 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj58 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj58 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ugbH4/0NEDodc4sK/G6G+vGfqI4= 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ugbH4/0NEDodc4sK/G6G+vGfqI4='} 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e0ef541f7d45d9e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.362 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de46d5d3ffe8bfc4-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj87 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj87 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj87 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj87 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.364 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:jWG0LE8oAjYjNbD4BrYtlGLn+F4= 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:jWG0LE8oAjYjNbD4BrYtlGLn+F4='} 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008683ed196478a9be-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000babd2e9ac98b968a-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj116 2023-09-30T00:47:11.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj116 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj116 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.367 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj116 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:4opB2jlCZOlBP/JRJVTcczssziw= 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:4opB2jlCZOlBP/JRJVTcczssziw='} 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096565618a305d254-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.369 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c8c4363e740601-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj145 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj145 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj145 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj145 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:kjEGs4Dirv1OchCWpXAmENEiadI= 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:kjEGs4Dirv1OchCWpXAmENEiadI='} 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcfc04edd762a9f5-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 25 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc70947b34893ebe-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj31 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj31 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj31 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj31 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zPzWkmJlp3rHJG9wWiCWcHdDhQg= 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zPzWkmJlp3rHJG9wWiCWcHdDhQg='} 2023-09-30T00:47:11.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000958b14d965e1b84e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.376 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084f259ac71f38546-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj62 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj62 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj62 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.378 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj62 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:VJjOuOpePsYtoMIHm7aqTlQctrg= 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:VJjOuOpePsYtoMIHm7aqTlQctrg='} 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3c105077b61b45c-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbe3a2f46c3399cd-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj93 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj93 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj93 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj93 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Cc1K4ZIe1xgRux5FXs3ovky6HgU= 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Cc1K4ZIe1xgRux5FXs3ovky6HgU='} 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014634d24b52e9242-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.383 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000716183e72cb11250-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj124 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj124 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj124 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj124 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.385 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zaFaIdS845tn5V3w3I9HT959SS8= 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zaFaIdS845tn5V3w3I9HT959SS8='} 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009931a890b1a1212-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be4a6bad169fb908-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj155 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj155 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj155 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.388 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj155 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ZjhH4a+P0T14immorJyqo7/k2xM= 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjhH4a+P0T14immorJyqo7/k2xM='} 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a637e44425f97f57-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'dmuhsu-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket dmuhsu-74 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: tenant: 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: bucket name: dmuhsu-74 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: total entries: 30 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a57aecc1bcb22b5c-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj37 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj37 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj37 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.392 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj37 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bByR5LTIxOaSVz2xFRbTjjNIMlk= 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bByR5LTIxOaSVz2xFRbTjjNIMlk='} 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c56bd7b2c0900c16-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.394 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea297e0feb8462f3-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj74 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj74 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj74 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj74 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:miMmxuzbrCwo4PfAluxsJdf9jFY= 2023-09-30T00:47:11.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:miMmxuzbrCwo4PfAluxsJdf9jFY='} 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7d85144b65cc9a-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000467bb035aaed9edb-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj111 2023-09-30T00:47:11.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj111 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj111 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj111 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:6JsWMfCgf7hzfxDuDdBScoM1jcQ= 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6JsWMfCgf7hzfxDuDdBScoM1jcQ='} 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014402872af3ee0aa-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk= 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:3wbiLo+jLRGEBU29Xbeo2fZF4Wk=', 'Content-Length': '0'} 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ea6433bb077e6e7-0065176790-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj148 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj148 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj148 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:56 GMT 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj148 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1cs7JbQ498vi7xlZU0OPIZPbWTg= 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:56 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1cs7JbQ498vi7xlZU0OPIZPbWTg='} 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5f150af141037e-0065176790-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/ 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/ 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/ 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.404 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:57 GMT 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/ 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:wSOLnOLYQwxAyuE4MX42FmWyCPA= 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:10:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:wSOLnOLYQwxAyuE4MX42FmWyCPA=', 'Content-Length': '0'} 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a353cd9a987232e1-0065176791-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:10:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-74/obj185 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-74/obj185 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-74/obj185 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:10:57 GMT 2023-09-30T00:47:11.406 INFO:tasks.rgw_multisite_tests:/dmuhsu-74/obj185 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:TnZFYxzcvGM+6G+lxULBW+FEhRA= 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:10:57 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:TnZFYxzcvGM+6G+lxULBW+FEhRA='} 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000150c191bc658ed55-0065176791-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:10:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-74 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-74 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-74 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:10:57.213+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:02.214+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:07.215+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:12.217+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:17.217+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:22.218+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:27.219+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:32.219+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:37.220+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:42.221+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:47.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:52.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:11:57.222+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:02.223+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:07.224+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:12.225+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:17.226+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:22.227+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:27.228+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:32.228+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:37.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:42.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:47.230+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:52.229+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:12:57.230+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:02.231+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:07.232+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:12.233+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:17.233+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:22.234+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:27.235+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:32.236+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:37.237+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:42.238+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:47.239+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:52.238+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:13:57.239+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:02.240+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:07.241+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:12.242+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:17.243+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:22.244+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:27.244+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:32.245+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:37.245+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:42.246+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:47.247+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:52.246+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:14:57.247+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:02.248+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:07.249+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:12.250+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:17.251+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:22.252+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:27.252+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:32.253+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:37.253+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:42.254+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:47.254+0000 7fee76610880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:52.255+0000 7fee76610880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:52.255+0000 7fee76610880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:52.255+0000 7fee76610880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:11.413 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1882, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-30T00:47:11.414 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.416 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:52 GMT 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:EDdS5YPQ2eskQUNiw9mSb0YIalI= 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:EDdS5YPQ2eskQUNiw9mSb0YIalI=', 'Content-Length': '0'} 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000047b41f88e63588bf-00651768b8-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dmuhsu-75 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:15:52.302803Z" 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:15:52.825+0000 7f760af1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:15:52.826+0000 7f760af1f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.430 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:52.927+0000 7f959d810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:52.928+0000 7f959d810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032952.295269_705.1", 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:15:52.295269Z", 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:47:11.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.439 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.440 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032952.295269_705.1", 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:15:52.295269Z", 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.441 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.442 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:15:52Z" 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.295269_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.295269Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.295269_705.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696032952.302803_708.1 target=1_1696032952.295269_705.1 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.443 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:58.046+0000 7ff768267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:58.047+0000 7ff768267880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:15:58Z" 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.450 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.451 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.452 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1696032952.302803_708.1", 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:15:52.302803Z", 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.453 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.454 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:15:58Z" 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696032952.302803_708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:15:52.302803Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696032952.302803_708.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.456 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b491e6525b0b91c1-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/obj11 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/obj11 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/obj11 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.458 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/obj11 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:NSF7wt0nmeKCeSQG552Y17l2GlQ= 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:NSF7wt0nmeKCeSQG552Y17l2GlQ='} 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000379525ca63744858-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:47:11.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b33962402bdfeb86-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/obj22 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/obj22 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/obj22 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.461 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/obj22 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:WGJ2PxzxGaJn9e2eTwKCg83tJLw= 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:WGJ2PxzxGaJn9e2eTwKCg83tJLw='} 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046e1d0cb0b1f8afb-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.463 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057743ab241a3af66-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/obj33 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/obj33 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/obj33 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/obj33 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:0PBzzvuWax3BsfHbNIynADnIa4g= 2023-09-30T00:47:11.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:0PBzzvuWax3BsfHbNIynADnIa4g='} 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b5fd468f3678c44-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000944c3338dc3f5d22-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/obj44 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/obj44 2023-09-30T00:47:11.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/obj44 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.468 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/obj44 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:g7Eg8beNycVmEF8ruF8NbOb+g+c= 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:g7Eg8beNycVmEF8ruF8NbOb+g+c='} 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048815376a31914b8-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/ 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/ 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/ 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/ 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8= 2023-09-30T00:47:11.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:bqO4qztRAs31htSetQ+CbUiyOt8=', 'Content-Length': '0'} 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064d42cb03834aa8a-00651768be-1e9c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-75/obj55 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-75/obj55 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-75/obj55 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:15:58 GMT 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:/dmuhsu-75/obj55 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1yceiF7py6gt+AJIEfK+V2nbTK0= 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:15:58 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1yceiF7py6gt+AJIEfK+V2nbTK0='} 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d91f7979de6750c-00651768be-1e9c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:15:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=dmuhsu-75 2023-09-30T00:47:11.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: Sync is disabled for bucket dmuhsu-75 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=dmuhsu-75 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:15:58.383+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:03.385+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:08.386+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:13.387+0000 7f2eabf23880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:18.388+0000 7f2eabf23880 1 bucket sync caught up with source: 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: local status: [, , , , , , , , 00000000001.38.6, 00000000002.74.6, 00000000002.79.6] 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: remote markers: [, , , , , , , , 00000000001.38.6, 00000000002.74.6, 00000000002.79.6] 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:18.388+0000 7f2eabf23880 0 bucket checkpoint complete 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=dmuhsu-75 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dmuhsu-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-30T00:47:11.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dmuhsu-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:18.537+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:23.538+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:28.538+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:33.539+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:38.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:43.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:48.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:53.540+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:16:58.541+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:03.542+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:08.543+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:13.543+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:18.544+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:23.544+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:28.545+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:33.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:38.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:43.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:48.546+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:53.547+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:17:58.548+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:03.549+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:08.550+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:13.550+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:18.551+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:23.551+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:28.552+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:33.552+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:38.553+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:43.553+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:48.554+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:53.554+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:18:58.556+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:03.556+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:08.557+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:13.557+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:18.558+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:23.558+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:28.559+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:33.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:38.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:43.560+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:48.561+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:53.561+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:19:58.563+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:03.563+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:08.564+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:13.564+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:18.565+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:23.566+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:28.566+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:33.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:38.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:43.567+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:48.568+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:53.568+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:20:58.569+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:21:03.570+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:21:08.571+0000 7f273308b880 1 waiting to reach incremental sync.. 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:21:13.571+0000 7f273308b880 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:21:13.571+0000 7f273308b880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:21:13.571+0000 7f273308b880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.480 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2023-09-30T00:47:11.481 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.481 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.481 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2548, in test_sync_different_buckets 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:36:41.231011Z" 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:41.567+0000 7f37c751c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:41.568+0000 7f37c751c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:36:41Z" 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.491 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.492 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.493 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.494 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.495 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:36:41Z" 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:47:11.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:46.686+0000 7f53af723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:46.687+0000 7f53af723880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:36:46Z" 2023-09-30T00:47:11.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.503 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.504 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.505 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.506 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:36:46Z" 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:51.801+0000 7fd199f27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:36:51.802+0000 7fd199f27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.518 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.519 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034201.231011_1059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:36:41.231011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.520 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:36:51.916+0000 7f197d733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:36:51.917+0000 7f197d733880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.527 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.528 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.529 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.530 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.531 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.532 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.532 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:36:51Z" 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:47:11.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:36:57.055+0000 7f1a5fe67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:36:57.056+0000 7f1a5fe67880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:36:57Z" 2023-09-30T00:47:11.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.540 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034077.470578_1054.1", 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.470578Z", 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.544 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:36:57Z" 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034077.470578_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.470578Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034077.470578_1054.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034201.231011_1059.1 target=1_1696034077.470578_1054.1 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:02.185+0000 7f13a5467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:02.187+0000 7f13a5467880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:47:11.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:47:11.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.552 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.553 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.554 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034201.231011_1059.1", 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:36:41.231011Z", 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.555 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.556 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034201.231011_1059.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:36:41.231011Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034201.231011_1059.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[]} 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-85 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/ 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/ 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/ 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.559 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:02 GMT 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/ 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:FKD/l7LMHiS+T9mHn/VyxGdE6bI= 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FKD/l7LMHiS+T9mHn/VyxGdE6bI=', 'Content-Length': '0'} 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003b215159a44351f7-0065176dae-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-86 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-86/ 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-86/ 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-86/ 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:02 GMT 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:/dmuhsu-86/ 2023-09-30T00:47:11.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:8jBjkCJyCOrzg0N4F4HsZOS9avE= 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8jBjkCJyCOrzg0N4F4HsZOS9avE=', 'Content-Length': '0'} 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fb7bf512a0416496-0065176dae-2585-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:02.597+0000 7f6b69d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:02.598+0000 7f6b69d36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.354141_1060.1", 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.354141Z", 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.570 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.571 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.354141_1060.1", 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.354141Z", 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.575 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:02Z" 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.354141_1060.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.354141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.354141_1060.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034222.397146_1067.1 target=1_1696034222.354141_1060.1 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:07.708+0000 7fadcb736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:07.709+0000 7fadcb736880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.582 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.584 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.587 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:07.834+0000 7f599d929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:07.835+0000 7f599d929880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.595 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.595 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.595 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.598 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.598 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.599 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.601 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.602 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:07Z" 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:47:11.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-85', '--dest-bucket=dmuhsu-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-85', '--dest-bucket=dmuhsu-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-85 --dest-bucket=dmuhsu-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "epoch": 17, 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": [], 2023-09-30T00:47:11.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_map": { 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zonegroups": [ 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "test-zonegroup", 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zonegroup", 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "api_name": "", 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "is_master": true, 2023-09-30T00:47:11.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:11.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:11.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames": [], 2023-09-30T00:47:11.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames_s3website": [], 2023-09-30T00:47:11.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone3", 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:11.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:11.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:11.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:11.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:11.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:11.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone1", 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:11.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:11.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone2", 2023-09-30T00:47:11.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:11.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": false, 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "placement_targets": [ 2023-09-30T00:47:11.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "default-placement", 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [], 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "storage_classes": [ 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "STANDARD" 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "default_placement": "default-placement", 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_policy": { 2023-09-30T00:47:11.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "groups": [ 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-group", 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_flow": { 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "symmetrical": [ 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-flow1", 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:11.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pipes": [ 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-pipe", 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:11.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-85", 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": { 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-86", 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "params": { 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "filter": { 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [] 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": {}, 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "priority": 0, 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mode": "system", 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user": "" 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "enabled" 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled_features": [ 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "short_zone_ids": [ 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2885950644 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:11.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2464975307 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 1018467004 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zonegroup": "test-zonegroup", 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_config": { 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_quota": { 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:11.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_quota": { 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_ratelimit": { 2023-09-30T00:47:11.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_ratelimit": { 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:11.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "anonymous_ratelimit": { 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_name": "test-realm", 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/ 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/ 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/ 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.632 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:08 GMT 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/ 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg= 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg=', 'Content-Length': '0'} 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054a8ecc3e9fe1a17-0065176db4-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/obj1 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/obj1 2023-09-30T00:47:11.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/obj1 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.634 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:08 GMT 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/obj1 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:HkHgE8LO3YC73om3Zwp6NlOOqYc= 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:HkHgE8LO3YC73om3Zwp6NlOOqYc='} 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e1d3b642a0b012-0065176db4-2658-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/ 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/ 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/ 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.636 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:08 GMT 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/ 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg= 2023-09-30T00:47:11.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Mp3of/2k6ERGGKZ+7sf/7LdFwTg=', 'Content-Length': '0'} 2023-09-30T00:47:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdc36e07fbcf49dc-0065176db4-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/obj2 2023-09-30T00:47:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/obj2 2023-09-30T00:47:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/obj2 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:37:08 GMT 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/obj2 2023-09-30T00:47:11.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:avICBwO7NODE2Jiju4myZgUB2o4= 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:37:08 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:avICBwO7NODE2Jiju4myZgUB2o4='} 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f25d9f4d77987fa4-0065176db4-2658-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:37:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:09.070+0000 7f3af1024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:37:09.071+0000 7f3af1024880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.651 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.656 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:09.243+0000 7f6ebc136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:37:09.244+0000 7f6ebc136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.669 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.670 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.671 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.674 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.683 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.684 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.685 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.686 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:09Z" 2023-09-30T00:47:11.688 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:14Z" 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.695 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.697 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:14Z" 2023-09-30T00:47:11.700 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:19Z" 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.707 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.710 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.711 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.712 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:19Z" 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:24Z" 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.720 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.722 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.723 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.724 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:24Z" 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:30Z" 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.732 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.733 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.734 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.735 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.738 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.738 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.739 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:30Z" 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:35Z" 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.746 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.748 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.750 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.751 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:35Z" 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:40Z" 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.759 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.762 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.763 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:40Z" 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:45Z" 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.771 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.773 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.774 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.775 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:45Z" 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:50Z" 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.783 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.784 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.785 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.786 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.787 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:50Z" 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:37:55Z" 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.795 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.796 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.797 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.798 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.799 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:37:55Z" 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:00Z" 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.806 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.807 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.811 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:00Z" 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:05Z" 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.819 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.820 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.823 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:05Z" 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:10Z" 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.830 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.832 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.833 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:10Z" 2023-09-30T00:47:11.835 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:16Z" 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.845 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.846 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.847 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:16Z" 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:21Z" 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.857 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.857 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.857 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.857 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.857 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.858 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.859 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.859 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.859 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.860 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.860 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.860 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.860 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.860 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.861 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.861 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.861 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.862 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.862 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.862 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.862 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.864 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.864 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.864 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.864 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.865 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.865 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.865 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.866 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.866 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.866 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.867 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:21Z" 2023-09-30T00:47:11.868 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.868 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:26Z" 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.880 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.885 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:26Z" 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:31Z" 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.893 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.894 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.895 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.896 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.897 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:31Z" 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:36Z" 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.905 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024227", 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:34:37.712703Z" 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.907 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.908 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018334", 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.234852Z" 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.909 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:36Z" 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024227', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:34:37.712703Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.234852Z'}}] 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024227', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018334'} 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000024367 target=00000000000000000000:00000000000000024227, shard=3 master=00000000000000000000:00000000000000018474 target=00000000000000000000:00000000000000018334 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:11.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:11.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.917 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:11.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:11.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.922 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/ 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/ 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/ 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:41 GMT 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/ 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.924 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mPQAIgjv9iHvkgLlFQlsHOZfIOM= 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mPQAIgjv9iHvkgLlFQlsHOZfIOM=', 'Content-Length': '0'} 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aa844018f712c65-0065176e11-1f62-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/obj1 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/obj1 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/obj1 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:41 GMT 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/obj1 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:6/ygMYOt41dKMg54L2Is1g6Y/GU= 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6/ygMYOt41dKMg54L2Is1g6Y/GU=', 'Content-Length': '0'} 2023-09-30T00:47:11.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a873de4c7c422197-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-85/obj2 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-85/obj2 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-85/obj2 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.927 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:41 GMT 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:/dmuhsu-85/obj2 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:K6mXwRXwXSISX5VgJSGjWw6FvYg= 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:K6mXwRXwXSISX5VgJSGjWw6FvYg=', 'Content-Length': '0'} 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000084504d5956cd39fc-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-86/ 2023-09-30T00:47:11.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-86/ 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-86/ 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.929 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:41 GMT 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:/dmuhsu-86/ 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:QtdIw2gSXEfqr3wktxZvzuzhT7M= 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:QtdIw2gSXEfqr3wktxZvzuzhT7M=', 'Content-Length': '0'} 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032834f77cba017cc-0065176e11-1f62-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-86/obj1 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-86/obj1 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-86/obj1 2023-09-30T00:47:11.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:41 GMT 2023-09-30T00:47:11.931 INFO:tasks.rgw_multisite_tests:/dmuhsu-86/obj1 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ndzHa0U2dnJaPfqkvy0rSdgKjwc= 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:41 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ndzHa0U2dnJaPfqkvy0rSdgKjwc=', 'Content-Length': '0'} 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cb081bf9d6ac6503-0065176e11-1f62-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:38:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-09-30T00:47:11.932 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:11.934 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-30T00:47:11.934 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-30T00:47:11.934 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-30T00:47:11.934 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2745, in test_sync_multiple_buckets_to_single 2023-09-30T00:47:11.934 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesD, content) 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:37:02.397146Z" 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:41.906+0000 7f8c5ec26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:41.907+0000 7f8c5ec26880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:47:11.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.948 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.948 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.950 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.951 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.952 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.952 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.954 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:41Z" 2023-09-30T00:47:11.956 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:38:42.028+0000 7f63ff017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:38:42.029+0000 7f63ff017880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.966 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.969 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-87 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:11.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:42 GMT 2023-09-30T00:47:11.972 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:SpbZ5E5cM00GG/T7vOQu7WlctIQ= 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:SpbZ5E5cM00GG/T7vOQu7WlctIQ=', 'Content-Length': '0'} 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000510d2902a9cb7bf4-0065176e12-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=dmuhsu-88 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:11.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.974 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:38:42 GMT 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Eof8SRN4DPl/AamT6OB0sWJ51Uw= 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:38:42 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Eof8SRN4DPl/AamT6OB0sWJ51Uw=', 'Content-Length': '0'} 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002404ae5f4e95acb3-0065176e12-2658-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:38:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:11.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:38:42.186786Z" 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:11.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:11.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:11.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:11.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:42.367+0000 7fbdf551e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:42.368+0000 7fbdf551e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:11.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:11.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:11.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:11.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:11.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:11.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:11.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:11.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:11.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:11.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.993 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:11.993 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.993 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:11.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.994 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.995 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:11.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:11.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:11.998 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:42Z" 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:11.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:47.478+0000 7f12b7729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:47.479+0000 7f12b7729880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:47Z" 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.006 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.007 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.010 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.012 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:47Z" 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.013 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:12.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:52.591+0000 7f504540a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:52.592+0000 7f504540a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:52Z" 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.024 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.025 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.027 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.028 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:52Z" 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:57.704+0000 7f9285333880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:38:57.705+0000 7f9285333880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:38:57Z" 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.036 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.037 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.040 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:38:57Z" 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:02.818+0000 7fe93e80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:02.819+0000 7fe93e80a880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:02Z" 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.048 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.053 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:02Z" 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.054 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:07.934+0000 7f8ca1414880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:07.935+0000 7f8ca1414880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:07Z" 2023-09-30T00:47:12.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.062 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034222.397146_1067.1", 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:02.397146Z", 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.065 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:07Z" 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034222.397146_1067.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:02.397146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034222.397146_1067.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034222.397146_1067.1 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:13.047+0000 7fd324d27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:13.048+0000 7fd324d27880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.163175_1099.1", 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.163175Z", 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:13Z" 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.073 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.163175_1099.1", 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.163175Z", 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.077 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.078 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:13Z" 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.163175_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.163175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.163175_1099.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034322.186786_1106.1 target=1_1696034322.163175_1099.1 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:18.167+0000 7f73edd1c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:18.169+0000 7f73edd1c880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.086 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.091 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:39:18.279+0000 7f3000a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:39:18.280+0000 7f3000a10880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.099 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.100 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.102 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:18Z" 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.105 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-87', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-87', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-87 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-87 --dest-bucket=dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-88', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=dmuhsu-88', '--dest-bucket=dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dmuhsu-88 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=dmuhsu-88 --dest-bucket=dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "epoch": 18, 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": [], 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_map": { 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zonegroups": [ 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "test-zonegroup", 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zonegroup", 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "api_name": "", 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "is_master": true, 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames": [], 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames_s3website": [], 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone3", 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone1", 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:12.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone2", 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": false, 2023-09-30T00:47:12.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "placement_targets": [ 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "default-placement", 2023-09-30T00:47:12.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [], 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "storage_classes": [ 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "STANDARD" 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "default_placement": "default-placement", 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_policy": { 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "groups": [ 2023-09-30T00:47:12.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-group", 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_flow": { 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "symmetrical": [ 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-flow1", 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "directional": [ 2023-09-30T00:47:12.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest_zone": "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pipes": [ 2023-09-30T00:47:12.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-pipe", 2023-09-30T00:47:12.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-85", 2023-09-30T00:47:12.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": { 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-86", 2023-09-30T00:47:12.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "params": { 2023-09-30T00:47:12.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "filter": { 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [] 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": {}, 2023-09-30T00:47:12.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "priority": 0, 2023-09-30T00:47:12.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mode": "system", 2023-09-30T00:47:12.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user": "" 2023-09-30T00:47:12.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-pipe-dmuhsu-87", 2023-09-30T00:47:12.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-87", 2023-09-30T00:47:12.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": { 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-88", 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "params": { 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "filter": { 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [] 2023-09-30T00:47:12.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": {}, 2023-09-30T00:47:12.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "priority": 0, 2023-09-30T00:47:12.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mode": "system", 2023-09-30T00:47:12.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user": "" 2023-09-30T00:47:12.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-pipe-dmuhsu-88", 2023-09-30T00:47:12.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-88", 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e" 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": { 2023-09-30T00:47:12.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-88", 2023-09-30T00:47:12.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "params": { 2023-09-30T00:47:12.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "filter": { 2023-09-30T00:47:12.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [] 2023-09-30T00:47:12.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": {}, 2023-09-30T00:47:12.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "priority": 0, 2023-09-30T00:47:12.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mode": "system", 2023-09-30T00:47:12.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user": "" 2023-09-30T00:47:12.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "enabled" 2023-09-30T00:47:12.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled_features": [ 2023-09-30T00:47:12.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "short_zone_ids": [ 2023-09-30T00:47:12.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:12.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2885950644 2023-09-30T00:47:12.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2464975307 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 1018467004 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zonegroup": "test-zonegroup", 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_config": { 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_quota": { 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:12.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_quota": { 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:12.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_ratelimit": { 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_ratelimit": { 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "anonymous_ratelimit": { 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_name": "test-realm", 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.149 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw= 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw=', 'Content-Length': '0'} 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd25b6f0398fdcb-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj1 2023-09-30T00:47:12.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj1 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj1 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.151 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj1 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:1tQk14w0DntvCXU9mUX9I926QEY= 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:1tQk14w0DntvCXU9mUX9I926QEY='} 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b93dc08825624c28-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.153 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw= 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ru+zUBqD95ghygHA2FEwiNnFFlw=', 'Content-Length': '0'} 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6edd77e9608cee4-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj2 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj2 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj2 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj2 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:H9ZF2ZRYR4nXJa7hLuPIC7jvcho= 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:H9ZF2ZRYR4nXJa7hLuPIC7jvcho='} 2023-09-30T00:47:12.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000526e2252af677e7e-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU= 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU=', 'Content-Length': '0'} 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9a46fbb2eb7b840-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj3 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj3 2023-09-30T00:47:12.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj3 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.158 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj3 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:2tDF0guenHidnmhEZKXyosteqTI= 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:2tDF0guenHidnmhEZKXyosteqTI='} 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000892ecedf1b6641b7-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.160 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU= 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ZjKrW7ZUmyuwO2zwN0TA88JRViU=', 'Content-Length': '0'} 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e1dc089ae220f7f-0065176e37-26da-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj4 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj4 2023-09-30T00:47:12.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj4 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:39:19 GMT 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj4 2023-09-30T00:47:12.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:n71fatriBdqpSdSf/MhFwqqLSMY= 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:39:19 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:n71fatriBdqpSdSf/MhFwqqLSMY='} 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e09249a3c60b927-0065176e37-26da-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:12.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:38:42.186786Z" 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:19.794+0000 7fbeb400f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:39:19.795+0000 7fbeb400f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.177 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.178 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.179 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.180 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.181 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:39:19.939+0000 7fb0ad726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:39:19.940+0000 7fb0ad726880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.195 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:19Z" 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.196 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:12.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:20Z" 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.208 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.209 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.210 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.213 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:20Z" 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.214 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:25Z" 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.220 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.221 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.222 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.223 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.225 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:25Z" 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:30Z" 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.234 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.236 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.237 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:30Z" 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:35Z" 2023-09-30T00:47:12.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.249 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.249 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.249 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.249 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.249 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.250 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.253 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.254 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:35Z" 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:40Z" 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.261 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.262 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.266 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:40Z" 2023-09-30T00:47:12.267 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:45Z" 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.278 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.279 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.280 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.281 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.282 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.283 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:45Z" 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:50Z" 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.291 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.292 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.293 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.294 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.295 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.296 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:50Z" 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.297 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:39:56Z" 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.304 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.305 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.305 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.306 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.306 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.306 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.306 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.306 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.307 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.307 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.307 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.308 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.309 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.309 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.311 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.312 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:39:56Z" 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:01Z" 2023-09-30T00:47:12.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.324 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.325 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.326 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.327 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.328 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:01Z" 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:06Z" 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.336 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.337 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.338 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.339 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.340 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:06Z" 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:11Z" 2023-09-30T00:47:12.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.349 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.349 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.349 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.349 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.351 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.354 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:11Z" 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.355 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:16Z" 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.365 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.370 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:16Z" 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:21Z" 2023-09-30T00:47:12.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.377 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.378 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.379 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.381 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:21Z" 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:26Z" 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.391 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.392 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.394 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.395 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.396 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:26Z" 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:31Z" 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.404 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.407 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.408 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:31Z" 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:36Z" 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.415 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.417 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.417 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.417 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.417 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.418 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.420 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.421 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:36Z" 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:42Z" 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.432 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.433 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.434 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.437 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:42Z" 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:47Z" 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.444 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.445 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013370", 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:31:43.024769Z" 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024367", 2023-09-30T00:47:12.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.718748Z" 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.447 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.448 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018474", 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:37:08.745804Z" 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:47Z" 2023-09-30T00:47:12.450 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013370', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:31:43.024769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024367', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.718748Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018474', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:37:08.745804Z'}}] 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013370', 1: '00000000000000000000:00000000000000024367', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018474'} 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013510 target=00000000000000000000:00000000000000013370, shard=1 master=00000000000000000000:00000000000000024647 target=00000000000000000000:00000000000000024367, shard=3 master=00000000000000000000:00000000000000018614 target=00000000000000000000:00000000000000018474 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:52Z" 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.457 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.458 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.461 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:52Z" 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:hhzzcAiiNPWZlBaIVKQ6LIvQgPE= 2023-09-30T00:47:12.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:hhzzcAiiNPWZlBaIVKQ6LIvQgPE=', 'Content-Length': '0'} 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da07c63280ae923a-0065176e94-204e-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj1 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj1 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj1 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj1 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.466 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:qplxDd7l6XiXpcrCytV3S7OC7wk= 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:qplxDd7l6XiXpcrCytV3S7OC7wk=', 'Content-Length': '0'} 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e599d5c116dd4a6f-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj2 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj2 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj2 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.468 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj2 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:Y8midTiPqzUuQxAfmLua2gLzRVA= 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:Y8midTiPqzUuQxAfmLua2gLzRVA=', 'Content-Length': '0'} 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000022bb366fd6fd4a6-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj3 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj3 2023-09-30T00:47:12.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj3 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.470 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj3 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:d6tbx1XqAv1JqRoqrK9vXvGf9Uk= 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d6tbx1XqAv1JqRoqrK9vXvGf9Uk=', 'Content-Length': '0'} 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004082988b79d081f1-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj4 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj4 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj4 2023-09-30T00:47:12.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj4 2023-09-30T00:47:12.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:+klwmmxzxhyrlzdJq/dBhF8P0f8= 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:+klwmmxzxhyrlzdJq/dBhF8P0f8=', 'Content-Length': '0'} 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d7221e21254b4758-0065176e94-204e-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:8PB+5srOv7Zcljx13TZLwjzi9SY= 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:8PB+5srOv7Zcljx13TZLwjzi9SY=', 'Content-Length': '0'} 2023-09-30T00:47:12.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008872d1ae327efdcc-0065176e94-204e-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj1 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj1 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj1 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj1 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:zoZxqcTVahoNJg2sCuuoqrbUwMI= 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:zoZxqcTVahoNJg2sCuuoqrbUwMI=', 'Content-Length': '0'} 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051a5820fcb6722f8-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj1 2023-09-30T00:47:12.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj1 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj1 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests:GET 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.478 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.479 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj1 2023-09-30T00:47:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.479 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:6rjOnxyBhkUhjUnE8wNLsMB/ieA= 2023-09-30T00:47:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:6rjOnxyBhkUhjUnE8wNLsMB/ieA=', 'Content-Length': '0'} 2023-09-30T00:47:12.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003027ab0e380050bc-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj2 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj2 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj2 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj2 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:BhHvh7zoXYpTgmFOJXFvvDWDsCc= 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:BhHvh7zoXYpTgmFOJXFvvDWDsCc=', 'Content-Length': '0'} 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014ef7a7e98e87b30-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj2 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj2 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj2 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:GET 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj2 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:K0ERW6p44nP95mF136gmHv+bJGo= 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:K0ERW6p44nP95mF136gmHv+bJGo=', 'Content-Length': '0'} 2023-09-30T00:47:12.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-87:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d9506aeef81343b-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj3 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj3 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj3 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.485 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.486 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.486 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.486 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj3 2023-09-30T00:47:12.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.486 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:CVxq+r3s1Mxs8MiOrRNKPFfG2mc= 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:CVxq+r3s1Mxs8MiOrRNKPFfG2mc=', 'Content-Length': '0'} 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034edf68711f1119c-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj3 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj3 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-30T00:47:12.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj3 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests:GET 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj3 2023-09-30T00:47:12.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:vFV3TOc+PlImxp99ZDDsXaRrUAQ= 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vFV3TOc+PlImxp99ZDDsXaRrUAQ=', 'Content-Length': '0'} 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000575a3ecf6e3bb6ad-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj4 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj4 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj4 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj4 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.491 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:c5S+EoqQVCkDR3F1G04NneZ9ETk= 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:c5S+EoqQVCkDR3F1G04NneZ9ETk=', 'Content-Length': '0'} 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006552e27ab3dbd715-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj4 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj4 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj4 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:GET 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:40:52 GMT 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj4 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:yU7+8Ugtzp38tQktCoz/rZsAeeM= 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:40:52 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:yU7+8Ugtzp38tQktCoz/rZsAeeM=', 'Content-Length': '0'} 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 30 Sep 2023 00:39:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'd5be89e5-4ad9-4a27-8129-17eff73a6a3e:dmuhsu-88:d5be89e5-4ad9-4a27-8129-17eff73a6a3e.9818.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008262c07439094ed0-0065176e94-204e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 30 Sep 2023 00:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-88","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dmuhsu-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dmuhsu-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "epoch": 19, 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "predecessor_uuid": "2bdb37d0-ae81-48f2-8aab-f47771c96a40", 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": [], 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_map": { 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zonegroups": [ 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "test-zonegroup", 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zonegroup", 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "api_name": "", 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "is_master": true, 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames": [], 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "hostnames_s3website": [], 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone3", 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8001" 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone1", 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi175.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": true, 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "test-zone2", 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "endpoints": [ 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "http://smithi184.front.sepia.ceph.com:8000" 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_meta": false, 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "log_data": true, 2023-09-30T00:47:12.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "read_only": false, 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tier_type": "", 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from_all": false, 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_from": [], 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "redirect_zone": "", 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "supported_features": [ 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "compress-encrypted", 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "placement_targets": [ 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "name": "default-placement", 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [], 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "storage_classes": [ 2023-09-30T00:47:12.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "STANDARD" 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "default_placement": "default-placement", 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_policy": { 2023-09-30T00:47:12.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "groups": [ 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-group", 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "data_flow": { 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "symmetrical": [ 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-flow1", 2023-09-30T00:47:12.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pipes": [ 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "id": "sync-pipe", 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-85", 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": { 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket": "dmuhsu-86", 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "zones": [ 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "e97e2b17-58fb-4667-bf95-8d29031e4378" 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "params": { 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "source": { 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "filter": { 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "tags": [] 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "dest": {}, 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "priority": 0, 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "mode": "system", 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user": "" 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "allowed" 2023-09-30T00:47:12.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled_features": [ 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "resharding" 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ], 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "short_zone_ids": [ 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "4e047d8a-7fd9-483a-b356-0bddc8ecae27", 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2885950644 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 2464975307 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": "e97e2b17-58fb-4667-bf95-8d29031e4378", 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": 1018467004 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zonegroup": "test-zonegroup", 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "master_zone": "d5be89e5-4ad9-4a27-8129-17eff73a6a3e", 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period_config": { 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_quota": { 2023-09-30T00:47:12.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_quota": { 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false, 2023-09-30T00:47:12.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "check_on_raw": false, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size": -1, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_size_kb": 0, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_objects": -1 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "user_ratelimit": { 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "bucket_ratelimit": { 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "anonymous_ratelimit": { 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_ops": 0, 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_ops": 0, 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_read_bytes": 0, 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "max_write_bytes": 0, 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "enabled": false 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_id": "de44a8c3-8a59-46e4-824e-eaf8e5fcea33", 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_name": "test-realm", 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-30T00:47:12.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[],"status":"allowed"}]} 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-87', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.524 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-87', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-87', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-87 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-87 --source-bucket=dmuhsu-87 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-88', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dmuhsu-88', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--bucket=dmuhsu-87', '--source-bucket=dmuhsu-88', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dmuhsu-88 --source-zones=test-zone1 --dest-zones=test-zone2 --bucket=dmuhsu-87 --source-bucket=dmuhsu-88 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dmuhsu-87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dmuhsu-87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe-dmuhsu-87","source":{"bucket":"dmuhsu-87","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dmuhsu-88","source":{"bucket":"dmuhsu-88","zones":["test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone1","test-zone2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dmuhsu-85","zones":["test-zone1","test-zone2"]},"dest":{"bucket":"dmuhsu-86","zones":["test-zone1","test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:40:54.259118Z" 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:40:54.587+0000 7f92927a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:40:54.588+0000 7f92927a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:54Z" 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.538 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.539 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.543 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:54Z" 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:40:59.699+0000 7fdfb9c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:40:59.700+0000 7fdfb9c15880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:40:59Z" 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.556 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:40:59Z" 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:04.810+0000 7f3a20212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:04.811+0000 7f3a20212880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:41:04Z" 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.565 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.566 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.567 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.568 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.569 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:04Z" 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:09.921+0000 7fe5f231f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:09.923+0000 7fe5f231f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:41:09Z" 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.577 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.578 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.579 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.580 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.581 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:09Z" 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:15.033+0000 7f2e5de13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:15.034+0000 7f2e5de13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:41:15Z" 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.589 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:15Z" 2023-09-30T00:47:12.594 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:20.147+0000 7fe43bc0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:20.147+0000 7fe43bc0f880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:41:20Z" 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.601 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:20Z" 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.606 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:25.259+0000 7fc3ca530880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:41:25.260+0000 7fc3ca530880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.615 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.618 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:25.376+0000 7f704bc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:25.377+0000 7f704bc13880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:47:12.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.627 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.628 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.629 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.630 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:25Z" 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:30.500+0000 7f232f332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:30.501+0000 7f232f332880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:30Z" 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.640 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.641 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.642 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.643 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:30Z" 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:35.643+0000 7f30671dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:35.644+0000 7f30671dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:35Z" 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.652 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.656 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:35Z" 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:40.771+0000 7f204d811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:40.772+0000 7f204d811880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:40Z" 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.664 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.665 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.666 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.667 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.668 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:40Z" 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:45.893+0000 7fc6b3f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:45.894+0000 7fc6b3f36880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:45Z" 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.677 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.678 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:45Z" 2023-09-30T00:47:12.681 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:51.009+0000 7f5664410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:51.010+0000 7f5664410880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:51Z" 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.688 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.693 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:51Z" 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.694 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:56.151+0000 7f8b903dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:41:56.152+0000 7f8b903dc880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:41:56Z" 2023-09-30T00:47:12.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.705 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:41:56Z" 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:01.275+0000 7ff709533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:01.276+0000 7ff709533880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:01Z" 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.712 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.714 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.716 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.717 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:01Z" 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.718 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:06.401+0000 7fabb6ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:06.403+0000 7fabb6ac0880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:06Z" 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.725 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.728 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.729 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.730 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:06Z" 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:11.527+0000 7f23b392e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:11.528+0000 7f23b392e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:11Z" 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.738 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.739 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:11Z" 2023-09-30T00:47:12.743 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:16.648+0000 7f887512e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:16.650+0000 7f887512e880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:16Z" 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.750 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.755 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:16Z" 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:21.765+0000 7f5fa9711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:21.766+0000 7f5fa9711880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:21Z" 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.763 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:21Z" 2023-09-30T00:47:12.768 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:26.890+0000 7f4538531880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:26.891+0000 7f4538531880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:26Z" 2023-09-30T00:47:12.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.776 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.777 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.778 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.779 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.780 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:26Z" 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:32.005+0000 7ff0e4215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:32.006+0000 7ff0e4215880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:32Z" 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.788 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.789 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.792 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.793 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:32Z" 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:37.142+0000 7f1325daf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:37.143+0000 7f1325daf880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:37Z" 2023-09-30T00:47:12.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.801 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.802 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.803 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.804 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.805 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:37Z" 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:42.257+0000 7f73ff721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:42.258+0000 7f73ff721880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:42Z" 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:42Z" 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.818 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:47.402+0000 7f5695867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:47.403+0000 7f5695867880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:47Z" 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.825 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.826 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.830 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:47Z" 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:52.525+0000 7f8da3536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:52.526+0000 7f8da3536880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:52Z" 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.837 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034322.186786_1106.1", 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:38:42.186786Z", 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.842 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:52Z" 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034322.186786_1106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:38:42.186786Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034322.186786_1106.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034322.186786_1106.1 2023-09-30T00:47:12.843 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:57.649+0000 7f06eb810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:42:57.650+0000 7f06eb810880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034453.507756_1140.1", 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:40:53.507756Z", 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:42:57Z" 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.850 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.851 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034453.507756_1140.1", 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:40:53.507756Z", 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.854 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.855 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:42:57Z" 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034453.507756_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:53.507756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034453.507756_1140.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1696034454.259118_1179.1 target=1_1696034453.507756_1140.1 2023-09-30T00:47:12.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:43:02.787+0000 7f0d3e1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:43:02.788+0000 7f0d3e1a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:43:02Z" 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.868 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:02Z" 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.870 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks= 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks=', 'Content-Length': '0'} 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007020b373a549b524-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj5 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj5 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj5 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.872 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj5 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:mRxQ8kvYAp46oez4A91az5iK08c= 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:mRxQ8kvYAp46oez4A91az5iK08c='} 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe3908832bdb976a-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.874 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks= 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:vc60/tbsbH0qbZUl+GSehcTXcks=', 'Content-Length': '0'} 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009694bcaf238e7d95-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj6 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj6 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj6 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj6 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:FNQzx4jeLTg5C95g4AJyQO4m9Jw= 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:FNQzx4jeLTg5C95g4AJyQO4m9Jw='} 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f65e10a059b6bebd-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.877 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o= 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o=', 'Content-Length': '0'} 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfb2d1f86b25ad27-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj7 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj7 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj7 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.879 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj7 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:oy40ckbj819Erg+Cc0JSh/p3dks= 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:oy40ckbj819Erg+Cc0JSh/p3dks='} 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d87e54f7ba02e63e-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o= 2023-09-30T00:47:12.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:/ZCpAbY8IB8vqoWgmrMhQ3n148o=', 'Content-Length': '0'} 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7ede8e1491690f-0065176f16-273c-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj8 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj8 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj8 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi175.front.sepia.ceph.com:8000 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:PUT 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:02 GMT 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj8 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:sI2PYCa0xbxP/a53TECnSwflCW0= 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 30 Sep 2023 00:43:02 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:sI2PYCa0xbxP/a53TECnSwflCW0='} 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004595785819cb1f68-0065176f16-273c-test-zone1'), ('Date', 'Sat, 30 Sep 2023 00:43:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: No --period given, using current period=6a43d71a-d207-46f3-bdd8-e7a78954a184 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.274277Z" 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T23:55:00.294167Z" 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:40:54.259118Z" 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-29T22:11:57.047167Z" 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:43:03.089+0000 7fb54b3a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: 2023-09-30T00:43:03.090+0000 7fb54b3a2880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "state": 1, 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 1, 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 1 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.892 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.893 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.897 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:43:03.193+0000 7fb5ef136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: 2023-09-30T00:43:03.194+0000 7fb5ef136880 20 RGW-SYNC:meta: read sync status 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "info": { 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "markers": [ 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 0, 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 1, 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 1, 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 2, 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "key": 3, 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "val": { 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "state": 1, 2023-09-30T00:47:12.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "realm_epoch": 2 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "total": 1, 2023-09-30T00:47:12.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "complete": 1 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "period": "6a43d71a-d207-46f3-bdd8-e7a78954a184", 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.274277_207.1", 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.274277Z", 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.906 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1696031700.294167_473.1", 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:55:00.294167Z", 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.907 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1696034454.259118_1179.1", 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:40:54.259118Z", 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.908 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "marker": "1_1696025517.047167_98.1", 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:11:57.047167Z", 2023-09-30T00:47:12.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696031700.274277_207.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.274277Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696031700.294167_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:55:00.294167Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1696034454.259118_1179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:40:54.259118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1696025517.047167_98.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:11:57.047167Z', 'realm_epoch': 2}}] 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696031700.274277_207.1', 1: '1_1696031700.294167_473.1', 2: '1_1696034454.259118_1179.1', 3: '1_1696025517.047167_98.1'} 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: [ 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: }, 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: { 2023-09-30T00:47:12.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: "last_update": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: } 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi175.stdout: INFO: ] 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.919 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.921 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.922 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.923 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:03Z" 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:43:08Z" 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.930 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.931 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.932 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.933 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.934 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:08Z" 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:43:13Z" 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.941 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013510", 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.485460Z" 2023-09-30T00:47:12.942 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024647", 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.452905Z" 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.943 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.944 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018614", 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:39:19.393348Z" 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.945 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:13Z" 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013510', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.485460Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024647', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.452905Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018614', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:39:19.393348Z'}}] 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013510', 1: '00000000000000000000:00000000000000024647', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018614'} 2023-09-30T00:47:12.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000013650 target=00000000000000000000:00000000000000013510, shard=1 master=00000000000000000000:00000000000000024927 target=00000000000000000000:00000000000000024647, shard=3 master=00000000000000000000:00000000000000018754 target=00000000000000000000:00000000000000018614 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "sync_status": { 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "info": { 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "sync", 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "num_shards": 4, 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "instance_id": 5247795594334234800 2023-09-30T00:47:12.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "markers": [ 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 0, 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 9, 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:47:12.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 1, 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 66, 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 2, 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: { 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "key": 3, 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "val": { 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "status": "incremental-sync", 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "next_step_marker": "", 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total_entries": 0, 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "pos": 0, 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: ] 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "full_sync": { 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "total": 75, 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "complete": 75 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: }, 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: "current_time": "2023-09-30T00:43:18Z" 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi184.stdout: INFO: } 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-30T00:47:12.952 INFO:tasks.rgw_multisite_tests: "instance_id": 5247795594334234800 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000013650", 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:43:02.909341Z" 2023-09-30T00:47:12.953 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024927", 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "total_entries": 66, 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:43:02.863822Z" 2023-09-30T00:47:12.954 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021828", 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:32:32.326318Z" 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: { 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018754", 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-30T00:43:02.851293Z" 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: } 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: ] 2023-09-30T00:47:12.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: "total": 75, 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: "complete": 75 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: }, 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-30T00:43:18Z" 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests:} 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013650', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.909341Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024927', 'next_step_marker': '', 'total_entries': 66, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.863822Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021828', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:32:32.326318Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-30T00:43:02.851293Z'}}] 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000013650', 1: '00000000000000000000:00000000000000024927', 2: '00000000000000000000:00000000000000021828', 3: '00000000000000000000:00000000000000018754'} 2023-09-30T00:47:12.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/ 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/ 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/ 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/ 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:5KPFrF4y5qlrBdxmkCLSlCcIehM= 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:5KPFrF4y5qlrBdxmkCLSlCcIehM=', 'Content-Length': '0'} 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000843500add6ad4dcd-0065176f26-20f2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj5 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj5 2023-09-30T00:47:12.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj5 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.960 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj5 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:E3CDrG0BZM7fCYuTynVGrlmTM0s= 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:E3CDrG0BZM7fCYuTynVGrlmTM0s=', 'Content-Length': '0'} 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cabb8fbb94c7df23-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj6 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj6 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj6 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj6 2023-09-30T00:47:12.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:nuFKUvu4OVFwgKNqgbXwzJI1MW8= 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:nuFKUvu4OVFwgKNqgbXwzJI1MW8=', 'Content-Length': '0'} 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b11772b5b07ccbcd-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj7 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj7 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj7 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj7 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.966 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:M19fHSLexUY9WnTveSS2gP0qCTg= 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:M19fHSLexUY9WnTveSS2gP0qCTg=', 'Content-Length': '0'} 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a695169c71ad56cc-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-88/obj8 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-88/obj8 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-88/obj8 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:/dmuhsu-88/obj8 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:XCN0AIeaubpPZ61x0Wu97JeNVVU= 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:XCN0AIeaubpPZ61x0Wu97JeNVVU=', 'Content-Length': '0'} 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0716a1fca1a738c-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/ 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/ 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/ 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.969 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/ 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:d0bjODNdvQHPDbBl20YNQxKY/bY= 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:d0bjODNdvQHPDbBl20YNQxKY/bY=', 'Content-Length': '0'} 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089ab83798fdd8660-0065176f26-20f2-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dmuhsu-87/obj7 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dmuhsu-87/obj7 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-30T00:47:12.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dmuhsu-87/obj7 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi184.front.sepia.ceph.com:8000 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:Sat, 30 Sep 2023 00:43:18 GMT 2023-09-30T00:47:12.971 INFO:tasks.rgw_multisite_tests:/dmuhsu-87/obj7 2023-09-30T00:47:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-30T00:47:12.972 INFO:tasks.rgw_multisite_tests:AWS PTXB0TDQIX657E2O5ERJ:ounF77NhOPISdLiMcgkUqw9HNOI= 2023-09-30T00:47:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Sat, 30 Sep 2023 00:43:18 GMT', 'Authorization': 'AWS PTXB0TDQIX657E2O5ERJ:ounF77NhOPISdLiMcgkUqw9HNOI=', 'Content-Length': '0'} 2023-09-30T00:47:12.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000e5489a1feddeaa92-0065176f26-20f2-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 30 Sep 2023 00:43:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-30T00:47:12.972 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-30T00:47:12.973 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.973 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-30T00:47:12.973 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 9324.715s 2023-09-30T00:47:12.973 INFO:tasks.rgw_multisite_tests: 2023-09-30T00:47:12.974 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, errors=1, failures=21) 2023-09-30T00:47:12.978 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-30T00:47:13.051 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=773c1f8257f44304a35f576dc56a8089 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-30T00:47:13.053 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-09-30T00:47:13.063 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-09-30T00:47:13.107 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-09-30T00:47:13.158 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-30T00:47:13.159 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:13.163 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-30T00:47:13.158+0000 7f305dfd3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 89389) UID: 0 2023-09-30T00:47:13.241 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2023-09-30T00:47:13.158+0000 7f3064412880 -1 shutting down 2023-09-30T00:47:13.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-09-30T00:47:19.261 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-30T00:47:19.262 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-09-30T00:47:19.318 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/vault-root-token 2023-09-30T00:47:19.350 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-30T00:47:19.351 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:19.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-09-30T00:47:19.391 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-30T00:47:19.390+0000 7f2c07b69640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 140061) UID: 0 2023-09-30T00:47:19.392 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2023-09-30T00:47:19.390+0000 7f2c0e00e880 -1 shutting down 2023-09-30T00:47:24.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-30T00:47:24.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-09-30T00:47:25.452 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-30T00:47:25.453 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-09-30T00:47:25.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/vault-root-token 2023-09-30T00:47:25.540 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-30T00:47:25.541 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:25.542 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-30T00:47:25.541+0000 7f5a0c45c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 118280) UID: 0 2023-09-30T00:47:25.542 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2023-09-30T00:47:25.541+0000 7f5a1280b880 -1 shutting down 2023-09-30T00:47:30.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-30T00:47:30.883 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-30T00:47:31.643 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-30T00:47:31.643 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-09-30T00:47:31.661 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/vault-root-token 2023-09-30T00:47:31.733 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-09-30T00:47:31.734 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:31.775 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2023-09-30T00:47:31.774+0000 7fdbacadb640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 87422) UID: 0 2023-09-30T00:47:31.775 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2023-09-30T00:47:31.774+0000 7fdbb3010880 -1 shutting down 2023-09-30T00:47:36.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-09-30T00:47:36.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-30T00:47:37.836 INFO:tasks.rgw.c2.client.1:Stopped 2023-09-30T00:47:37.836 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-09-30T00:47:37.854 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/vault-root-token 2023-09-30T00:47:37.925 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:47:37.936 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-09-30T00:47:38.266 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:47:38.266 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:47:38.267 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-09-30T00:47:38.267 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:38.311 INFO:tasks.ceph.c2.osd.0:Stopped 2023-09-30T00:47:38.311 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-09-30T00:47:38.311 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:38.357 INFO:tasks.ceph.c2.osd.1:Stopped 2023-09-30T00:47:38.357 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-09-30T00:47:38.358 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:38.397 INFO:tasks.ceph.c2.osd.2:Stopped 2023-09-30T00:47:38.397 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:47:38.398 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-09-30T00:47:38.398 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:38.462 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-09-30T00:47:38.463 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:47:38.463 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-09-30T00:47:38.463 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:38.524 INFO:tasks.ceph.c2.mon.a:Stopped 2023-09-30T00:47:38.524 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:47:38.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-30T00:47:38.594 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-30T00:47:38.595 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-09-30T00:47:38.885 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-30T00:47:38.885 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-09-30T00:47:38.997 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi184.front.sepia.ceph.com 2023-09-30T00:47:38.997 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-09-30T00:47:39.055 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:47:39.056 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/data/c2.mon.a.tgz 2023-09-30T00:47:39.057 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2023-09-30T00:47:39.071 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.yWZk1syPI5 2023-09-30T00:47:39.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.yWZk1syPI5 2023-09-30T00:47:39.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.yWZk1syPI5 2023-09-30T00:47:39.306 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.yWZk1syPI5 is 364KB 2023-09-30T00:47:39.402 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.yWZk1syPI5 2023-09-30T00:47:39.416 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:47:39.416 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:47:39.419 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:47:39.483 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:47:39.493 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-09-30T00:47:39.906 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:47:39.906 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:47:39.907 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-09-30T00:47:39.907 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:39.982 INFO:tasks.ceph.c1.osd.0:Stopped 2023-09-30T00:47:39.983 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-09-30T00:47:39.983 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:40.048 INFO:tasks.ceph.c1.osd.1:Stopped 2023-09-30T00:47:40.049 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-09-30T00:47:40.049 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:40.124 INFO:tasks.ceph.c1.osd.2:Stopped 2023-09-30T00:47:40.124 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:47:40.125 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-09-30T00:47:40.125 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:40.152 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-09-30T00:47:40.152 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:47:40.152 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-09-30T00:47:40.152 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:47:40.218 INFO:tasks.ceph.c1.mon.a:Stopped 2023-09-30T00:47:40.219 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:47:40.219 DEBUG:teuthology.orchestra.run.smithi175:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-30T00:47:40.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-30T00:47:40.334 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-09-30T00:47:40.670 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-30T00:47:40.670 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-09-30T00:47:40.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi175.front.sepia.ceph.com 2023-09-30T00:47:40.762 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-09-30T00:47:40.835 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:47:40.836 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/data/c1.mon.a.tgz 2023-09-30T00:47:40.836 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2023-09-30T00:47:40.858 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.Ql2YRfcQ1N 2023-09-30T00:47:40.858 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.Ql2YRfcQ1N 2023-09-30T00:47:41.019 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.Ql2YRfcQ1N 2023-09-30T00:47:41.124 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.Ql2YRfcQ1N is 409KB 2023-09-30T00:47:41.219 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.Ql2YRfcQ1N 2023-09-30T00:47:41.234 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:47:41.234 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:47:41.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:47:41.307 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:47:41.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:47:41.342 DEBUG:teuthology.orchestra.run.smithi184:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:47:41.368 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:47:41.381 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:47:41.386 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:47:41.388 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi175/crash 2023-09-30T00:47:41.389 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:47:41.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-09-30T00:47:41.440 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi184/crash 2023-09-30T00:47:41.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:47:41.470 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:47:41.470 DEBUG:teuthology.orchestra.run.smithi175:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:47:41.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:53:59.750 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:53:59.750 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi175/log 2023-09-30T00:53:59.752 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:54:42.489 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi184/log 2023-09-30T00:54:42.490 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:54:56.356 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:54:56.367 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:54:56.367 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:54:56.370 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:54:56.512 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:54:56.522 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:54:56.523 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:54:56.526 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:54:56.540 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2023-09-30T00:54:56.540 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2023-09-30T00:54:56.544 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:54:56.544 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2023-09-30T00:54:56.545 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 18 -298us[ -298us] +/- 91ms 2023-09-30T00:54:56.545 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv02.front.sepia.ceph.com 3 7 377 94 -357us[ -424us] +/- 55ms 2023-09-30T00:54:56.545 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +680us[ +680us] +/- 77ms 2023-09-30T00:54:56.545 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:54:56.616 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:54:56.616 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2023-09-30T00:54:56.616 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -307us[ -345us] +/- 91ms 2023-09-30T00:54:56.616 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -354us[ -393us] +/- 56ms 2023-09-30T00:54:56.617 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +703us[ +666us] +/- 77ms 2023-09-30T00:54:56.617 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:54:56.618 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:54:56.627 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:54:56.628 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:54:56.665 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:54:56.806 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:54:56.900 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:54:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:56.935 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:54:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:56.969 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:54:57.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:54:57.132 INFO:teuthology.task.internal:Duration was 10497.649238 seconds 2023-09-30T00:54:57.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:54:57.191 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:54:57.194 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:54:57.201 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:54:57.248 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:54:57.265 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:54:57.717 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:54:57.717 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2023-09-30T00:54:57.717 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:54:57.751 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2023-09-30T00:54:57.751 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:54:57.774 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:54:57.775 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:57.795 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:57.850 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:54:57.851 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:54:57.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:54:57.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:54:58.014 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:54:58.015 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:54:58.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:54:58.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:54:58.095 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:58.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:58.136 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2023-09-30T00:54:58.145 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2023-09-30T00:54:58.174 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:58.225 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:58.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:54:58.252 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:54:58.253 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi175 2023-09-30T00:54:58.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:54:58.302 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303/remote/smithi184 2023-09-30T00:54:58.302 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:54:58.339 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:54:58.340 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:54:58.348 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:54:58.388 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:54:58.405 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:54:58.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:54:58.440 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:54:58.441 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:54:58.443 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:54:58.455 INFO:teuthology.orchestra.run.smithi175.stdout: 265356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:54 /home/ubuntu/cephtest 2023-09-30T00:54:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: 265981 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:54 /home/ubuntu/cephtest 2023-09-30T00:54:58.459 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:54:58.488 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:54:58.584 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:54:58.612 DEBUG:teuthology.nuke:shortname: smithi175 2023-09-30T00:54:58.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:54:58.636 DEBUG:teuthology.nuke:shortname: smithi184 2023-09-30T00:54:58.636 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:54:58.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:51:56.285915', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGg1Rj0/eERk8ONdHWi2h+CdqkStQ5MR5QKLDJW4g0aqKq3EScLjXaZ/6ZMMiGzDXOObRTOT0dqrZkf9wcsjTQk='} 2023-09-30T00:54:58.665 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:51:56.284120', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjK/BJMklETggsMW2i4eJ/4RHkZRjVcDsj06GMnEus1nm4X8DpnPcTSVd41NffPZrjXOA+ewpgniZENFjXO//c='} 2023-09-30T00:54:58.708 INFO:teuthology.orchestra.console.smithi175:Power off 2023-09-30T00:54:58.709 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:54:58.743 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off 2023-09-30T00:54:58.744 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:58.774 INFO:teuthology.orchestra.console.smithi184:Power off 2023-09-30T00:54:58.775 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:54:58.797 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2023-09-30T00:54:58.803 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2023-09-30T00:54:58.803 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:58.835 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2023-09-30T00:55:02.798 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:02.837 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:02.919 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2023-09-30T00:55:02.964 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2023-09-30T00:55:06.920 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:06.965 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:07.035 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2023-09-30T00:55:07.079 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2023-09-30T00:55:07.179 INFO:teuthology.orchestra.console.smithi184:Power off completed 2023-09-30T00:55:11.035 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:11.153 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off 2023-09-30T00:55:11.254 INFO:teuthology.orchestra.console.smithi175:Power off completed 2023-09-30T00:55:11.359 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} duration: 10497.649238348007 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=773c1f8257f44304a35f576dc56a8089 status: fail success: false 2023-09-30T00:55:11.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:55:11.437 INFO:teuthology.run:FAIL